Extracting WhatsApp message history

10 July 2020
For some time I have been wanting to archive messages from WhatsApp onto my desktop. One simple way of exporting message histories is the export chat option — under Settings → Chat History → Export Chat — and using Google Drive as the handler for the exported files, but this was no good for me as my chat history far exceeds what can be exported using this method. As a result my only real option was to get at the underlying message database directly, but one does not simply grab these off the handset via file-system access — the main issue is that the databases are encrypted and the system is designed to make the key hard to obtain. This article explains how I eventually managed to get at my history data.

The major motive for exporting chat histories is that I have doubts about the future trustworthiness of WhatsApp so I may well have to stop using it. In the past I was a Facebook messenger beta-tester but I stopped using it because I had issues with the permissions the app required when these were considered together with the terms & conditions of use. In short Facebook has a thoroughly rotten corporate culture that has taken the piss with user privacy, and for me any integration of WhatsApp into Facebook is an unacceptable step.

Downloading backup from Google Drive

WhatsApp backups on Google Drive are stored as hidden application data, and it is possible to access this data using Google APIs running on a desktop system. B16f00t's “whapa” is an open-source Python3 program that uses such APIs to, among other things, download the messages and media backed up by WhatsApp. This is done by the WhaGoDrive portion of the utility, using credentials specified in cfg/settings.cfg:

[auth] gmail = example@gmail.com passw = sekret devid = celnumbr = 447812345678

Note that celnumber is the number linked to WhatsApp and not any number you might have registered for password recovery with Google, and it needs the country code as the first digits. The documentation states that two-factor authentication needs to be disabled (I did not use 2FA at the time I tried all of this), and there might also be the need to allow access by new devices. The message database(s) themselves are encrypted, and since the key is not stored on Google Drive, this has to be obtained from elsewhere. Media files such as photographs and movies are not encrypted.

Getting the encryption key

From background reading the decryption key is generated on WhatsApp servers and sent to the handset on registration, although it is unclear whether this key remains the same for the life-time of the WhatsApp account or whether there are circumstances that a new key gets generated. However on a non-rooted device one does not simply grab the key with something like adb pull, as it is kept in a secure area that things like file managers cannot usually access.

The approach I used was to use the backup functionality of the ADB (Android Debug Bridge), which is part of the platform tools component of Android's SDK, to get a full backup of the device. The resulting backup.ab file — assuming that the backup was done without encryption — can be converted to a tar file by shaving off the existing header and inserting a new one. There are many ways this can be done, one of which is using the following Linux commands:

printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" > backup.tar tail +25c backup.ab >> backup.tar

The encryption key is stored as apps/com.whatsapp/f/key within the archive although this may vary between handsets — the backup I used was one I took of my old Samsung S4 shortly before I decommissioned it, which was running Android 4.4.2 at the time. A pit-fall I fell into is the backup command being non-functional in versions of the Android platform tools later then 23.0.1 which from recollection is due to an incompatibility with the version of Android.

Using this method is not an option with newer handsets, since ADB backup/restore mechanism is now deprecated. I have heard that Android 6 onwards does not include secure application data in backups, and the backup/restore does not work at all in Android 9 onwards. If you don't have access to an older handset you will need to use some other method fot getting the encryption key.