How to use the Data Interception Environment

Open Genymotion and add a device – there are a large selection of phones that you can choose from. For this tutorial we’re going to set up a virtual Pixel phone running Android 8.1. Once you pick the phone you want to choose, select it.

You can leave most of the settings the same but we’d strongly suggest dropping the number of processors and the memory size, as it will improve the performance of the phone, and make it less likely to stall.

Start the phone you created.

Use the icons down the side to install Open GApps or Open Google Apps – this will add the Play Store to the Genymotion phone. Restart the device.  

Sign in to a Google account, this will allow you to access the Play Store. You can then download any apps you want to test. We’d also strongly recommend installing a browser, such as Google Chrome.

Once the OVA is extracted double click on it, which will open it in VirtualBox. Some information about the DIE will pop up, click Import, then read and agree to the license terms.

Start the Data Interception Environment. A menu should pop up – this includes all the key set features that you might want to use. This This includes an option to configure a wireless adaptor so you can connect a real phone to the environment. As part of this you may wish to enable hostapd, which is a wireless access point daemon. You can also run an APK through apk-mitm, which allows it to be decompiled so you can remove certificate pinning. The key option here is to start mitmproxy, which allows you to wirelessly intercept secure sessions – which is the option you will likely use most often. There is also an option for showing the documentation. And you can update the environment through the link provided, although this functionality is relatively experimental.

Once you have all the apps you wanted to test installed, shut down the virtual phone and go to the settings in VirtualBox for the phone you’re using. The key one to change here is to change the network adaptor, on adaptor two, from NAT to an internal network. We called it ‘iNet’. You can leave the rest of the settings as they were. You can leave the rest of the settings as they were. You'll be able to see that the environment also has a secondary adaptor, also connected to iNET.

The reason for changing this network interface is so that all the data will be transferred across the internal network, so that the DIE will be able to intercept them.

Start the phone through Genymotion. If you start the phone through VirtualBox the phone will start but you won’t be able to use the interface. Once the phone has started, look at the network settings on the handset, it should be connected to Android wifi. And it should say, "Connected. No internet." This is a good sign.

You should, hopefully, see that it has an IP address starting 100.64 which means that the environment has set the IP rather than it being coming from a localized NAT device, as they usually start 192.168. 56. The environment always gives the IPs in the 100.64.s32 range.

Start collecting and analysing data

Start Mitmproxy. Now open the web viewer in the phone and look at mitmproxy – you should see the request coming in on the data interception environment. You can also summon instructions there as to how to set up the device, including information around if you want to use mitmproxy with devices running API level 24 or higher.

Open your web browser on the virtual phone and go to mitm.it and download the certificate. Once the certificate is downloaded it will appear in the download section of the files app.

Start your local terminal so you can install certificates into the system root store as outlined in the documentation on the GitHub and inside the DIE.

Run ADB devices to see which devices are available to connect to. There’s only one device available, which is the Genymotion device.

Then start ADB shell, which will allow you to access the local console of the Genymotion phone.

If you change the directory to SD card/download) (cd /sdcard/Download/ and then list the directory (ls -lah), you should see the mitmproxy certificate listed there.

For the time being you can close mitmproxy as you don’t need it.

Copy and paste the open SSL command from below, the documentation on GitHub or from the local documentation into the local console in the VM running the DIE.

sudo openssl x509 -inform PEM -subject_hash_old -in /root/.mitmproxy/mitmproxy-ca-cert.pem | head -1

This will return the name you will later need to give the file in Android for it to recognize the certificate authority.  This is how you’re going to become the local administrator on the virtual device.

When we copied the open SSL command into the console, we got the output C8750F0D. This is the name we'll have to give the certificate later when we copy it into the system root store on the Genymotion device.

Go back to your local console, which should still be running ADB shell

First execute su to become super user on the device, i.e. the local administrator.

Remount the partition system folder. This allows us to modify system data, which is usually read only.

mount -o remount,rw /system

If that command does not work, try the below.

mount -o remount,rw /

Copy the certificate from the downloads folder into the systems certificate folder.  You can use this in the console: cp /sdcard/Download/mitmproxy-ca-cert.pem /system/etc/security/cacerts/<NamefromOpenSSLOutput>.0

When copying the file, you need to use the name you retrieved earlier running from running the open SSL command, for us that would be C8750F0D. And then the extension is .0 – making the full title C8750F0D.0

You now need to change the file permissions, so copy this into the console: chmod 644 /system/etc/security/cacerts/<NamefromOpenSSLOutput>.0

And again, type in the certificate's name to replace <NamefromOpenSSLOutput>.

You also need to change who the file belongs to and make sure it belongs to root: chown root:root /system/etc/security/cacerts/NamefromOpenSSLOutput>.0

Now remount the file system read only: mount -o remount,ro /system

Reboot the device.

Exit the console and restart mitmproxy. You should see on the left that the device is now rebooting. Open mitmproxy in the web browser – you should be able to see a large amount of data that’s being collected. This is because as the device boots up it checks for internet access.

Look at the WiFi settings, it should now just say ‘connected’, unlike before where it said ‘connected, no internet’.

You can check that mitmproxy is installed correctly, by opening your browser and going to a website. You can go to the list of things being intercepted by mitmproxy and you’ll be able to see all the requests being made to the website.

Then, if click on the status bar, you should see that the certificate being presented is from mitmproxy, and if you look at the certificate information you should see that the assurer is mitmproxy.

Just for the sake of completeness, go to Privacy International's website not on the phone and you should see your browser isn't being intercepted. Again, look at the certificate information - this certificate is usually issued by DigiSure, Inc. If you look at the certificate it even includes all the information about PI.

You can now close your browser, and all the other applications you had open – you don’t need any at the moment.

Open the settings app and do a search for certificates and look at the trusted credentials section. Displays, which displays the trusted CAs and click on the system section. You can scroll down and see that the mitmproxy is a trusted system root.

Close mitmproxy and start a new session. Every time mitmproxy is restarted, the previous capture is saved to disk. If you head back to mitmproxy's webpage. You should see the capture is now clear as a new capture has been started. We would recommend always starting a new capture with every different application you wish to test.

Start testing apps

Open the first app you want to test.

If you click on ‘request’, you should be able to see what the original request was and what the response was.

For example, this is Google Firebase and we can see what token was used to authenticate with and what response was given for that Firebase installation.

In the bottom right-hand corner you can also find out information such as the SDK conversion of this version of Firebase Analytics and you can see things like remote requests.

In the bottom right-hand corner you can also find out information such as the SDK conversion of this version of Firebase Analytics and you can see things like remote requests. You can also see downloaded imagery such as that displayed in the top portion of the Met Office app – in this case the logo.

As you click things in the app, such as accepting the privacy policy, you should see more connections being made – some will be content, others metrics, advertising, or other third party services.

Below is an example of a fairly standard message sent by Facebook’s SDK to graph.facebook. You can see there's quite a lot of information you can glean from this message including the event that's being sent to Facebook along with information about the device, the application, the advertising ID, all of which can be cross referenced by Facebook to create or add to a profile.

The DIE can tell you what’s being sent from your device and where to, but it cannot tell you want the company is doing with that information, what processing they are doing, or whether they are sending that data on to third parties. Above you can see what the app is sending to Facebook, but it can’t tell you what Facebook is doing with that information.

When you’re done with the app – you can close your mitmproxy session. All captures are stored locally in privacy’s home folder, so the most recent captures can all be reloaded. If you want to analyse another app – start a new mitmproxy session.

A note on Certificate Pinning

One of the new features of the data interception environment is the inclusion of APK MITM, which allows the removal of certificate pins. Some large apps, like Facebook, banking apps, Twitter and some of the other apps that have large numbers of users, use a technique called certificate pinning, which means that they expect certain certificates to be presented by the remote side by making a connection. And if the, the certificate differs, then they will not make the secured connection.

What APK MITM allows is the removal of those pins so you can continue to do analysis in the data interception environment.

Further Information

PI has made this tool available to assist individuals, researchers and organisations in developing a technical understanding of how their own data is being captured, processed and transferred by applications to 3rd parties. It should be used to analyse data on a device, virtual copy of a device, or an app, which the user owns or is legally authorised to use.

The Data Interception Environment software is free to use in accordance with terms of the GNU General Public License. However, Privacy International’s brand, logo and name are copyrighted and protected from unauthorised use. PI cannot endorse any modifications that users or any other third parties make to PI’s Data Interception Environment and attribute to PI.

PI is a charity registered in England and Wales. We have developed our Data Interception Environment tool in accordance with the laws, regulations and standards that apply to us in England and Wales. Any individual, researcher or organisation who makes use of of PI’s Data Interception Environemnt is responsible for ensuring that they are acting lawfully and in accordance with the laws, regulations and standards which apply to them in the jurisdiction they are operating in or to which they are subject.

Enjoy the Data Interception Environment. If you've found the DIE useful, you can show your support by donating to make sure PI can continue to create things you find useful