2

So, I've decided to try and do some analysis on my WhatsApp chat history. For that purpose, the built-in export function would work quite well, but it is for some reason restricted to the latest 40.000 messages. Some of my chats go back way further than that, and I'd like to include those conversations in my dataset.

And that's where it gets complicated. Things I've tried so far:

  1. A variety of recovery tools (drfone, BackupToolForWhatsApp,...) that invariably failed to connect and/or demanded a lot of money and/or contained malware

  2. Finding the msgstore.db.crypt12 as a file on the phone and opening it as text with various apps (some people claimed that worked, it didn't though, everything just crashed or complained about the format)

  3. Copying the msgstore.db.crypt12 onto my computer and there decrypting it with the key file, which led to the next problem, obtaining said key file.

  4. A bunch of outdated and thus no longer functional WhatsApp key extractor tools which tried to either grab the backups off Google Drive, or temporarily install a legacy Version of WhatsApp with a vulnerability that allowed the key to be extracted. This failed because the tools haven't been updated in like three years and I'm not Java-savy enough to go digging in their source code myself. I actually downloaded a legacy APK to try and manually install that old WhatsApp version, but that only led to an error about the datetime of my phone (which was set to automatic and thus was correct) and no key.

  5. Rooting my phone in order to get access to the directory where the key is hopefully located. This failed, perhaps because I did the mistake of updating my firmware to the July 2019 patch level, and so no root exploits seemed to work

  6. Digging around in protected directories with tools that do not quite require root access. However, I wasn't able to locate anything that even looked like the place WhatsApp might store its key.

  7. Reading through lots of forum posts on the same issue without any discernible results.

I'm using a Samsung Galaxy S8, model number SM-G950F, German localization, with Android 9.

My question is simple: What else can I do? It can't be that it's literally impossible to dig at the messages which WhatsApp is happily storing and backing up. I'd even be willing to buy some software as long as it's reasonably priced and guaranteed to work.

Also, what directory would I be looking for to get that key, assuming I'd somehow get root access?

migrated from superuser.com Jul 22 at 10:59

This question came from our site for computer enthusiasts and power users.

  • How about exporting each contact for its own? Maybe 40.000 is sufficient. regarding 4. someone managed it on S10 with Pie 9.0 the pitfall is reboot required between uninstall (-k)/install. date can be set to past manually 5. should be possible in near future (assuming you already unlocked) With root access you don't need the key, there is unencrypted database too you can use. its all in /data/data/com.whatsapp – alecxs Jul 22 at 11:49
  • @alecxs Unfortunately, I had individual chats with significantly more than 40.000 messages. And I saw the forum post where someone managed to do it with Android 9, but following the same steps did not yield any results. – Marc Vaisband Jul 23 at 23:41
1

So, here's the solution I ended up using. Even though it's not ideal, it worked out, so I'll leave this here in case anyone else stumbles upon this.

The weak link in the chain is WhatsApp Web. Specifically, it's possible to write a tool that just scrapes the data WhatsApp Web receives, and someone on the Chrome Web Store did exactly that: The Backup WhatsApp Chats extension. It's like €3, so well within what I'd consider reasonable. However, I'd expect that anyone with more knowledge of coding than I have could write such a script themselves, because the messages in WhatsApp Web are apparently just displayed as plain HTML.

It's reasonably fast and extracts all the non-media data you might want in csv format that can be parsed pretty well by pandas.

I'll accept my own answer because this is a solution within the parameters I set out, but if someone knows a better way, feel free to post it and I'll shift the accept.

  • Does this actually work with chats with millions of messages over several years though? – Turkeyphant Jul 23 at 23:26
  • 1
    @Turkeyphant I've successfully done it with 391.000 messages stretching back to July 2015, so I at least have no reason to assume it would break at higher scales. The largest of those chats contained 128.095 messages, which took a few, but no more than twenty minutes to export fully. – Marc Vaisband Jul 23 at 23:36
  • Thanks. How long did that take? And do you see anything on the WhatApp Web screen while it scrapes? – Turkeyphant Jul 23 at 23:37
  • 1
    @Turkeyphant In total it took me an hour for all the chats maybe, and while it scrapes, there's an overlay on the screen that says "Downloading messages" and displays a count of messages downloaded. – Marc Vaisband Jul 23 at 23:39
  • 1
    @Turkeyphant Nope, I'm afraid with this method you have to wait for the program to finish exporting. – Marc Vaisband Jul 24 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.