Montoring network traffic on Android phone (for research user study)

Question

For context: We would like to analyze user mobile online (browsing) behavior in a user study. The study has been approved by our university's IRB, participants will have to give informed consent and will receive remuneration. Of course, monitoring network traffic is very intrusive, but we're sincerely not trying to be the bad guys here. For example, we would like to know if people get there information from social media, news sites (and from which one).

Over the last to weeks, I've explored some potential approaches. While I have some basic experiences with Android development, I'm far from an expert. Here is what I've tried and considered -- I would appreciate any feedback to see what I got wrong and/or what I'm missing:

Side note: we cannot expect study participants to root their phones :)

• Mock-up browser (simple app with a WebView and rudimentary browser features) or a modified fork of Chrome. Here, on the one hand, I cannot really estimate the required effort for both ideas. And a big issue is that most mobile online use is done through apps that often have their on WebView.

• Proxy. I set up a proxy with SSL support, so if the participants change their proxy setting and install our CA certificate, we would see their traffic. However, apps (e.g., Facebook) are not required to use the proxy settings, and since Android 7 (API 24) apps by default do not recognize user-installed certificates. It worked pretty well with an older Android 6.0 phone, but now it's only a limited option. We could still see, for example, that a request to facebook.com was made, but no longer the exact URL.

• App Activity Monitoring. Since some apps wouldn't have used the proxy anyway, I tried to at least estimate which app was in the foreground. To do this, I wrote a simple app that periodically checked which app had the highest usage in the last, say, 10 seconds using the UsageStatsManager. This seemed to work reasonably well.

• Local on-device VPN. Lastly, I've tried using a local VPN using the VpnService; I found and used example code I found here. On the plus side, even apps that ignore the proxy have to go through that local VPN. However, I face two big problems. Firstly, while it seems to work it's very unstable. Some websites won't load properly and after that, even sites that worked before fail. And secondly, there's no SLL support so I wouldn't be able to see the traffic. I lack the expertise right now to address both issues or even to say if they can be solved.

This is where we currently stand. All opinions are more than welcome.