This was a long process with lots of side problems coming up. Somethings I don't know why they worked or did not work. I will give a recount of what did work in hopes the information will be useful in the future.
At a high level the idea is to backup the data partition, reflash firmware and ROM to fix the bootloop, reload the backed up data partition. Perhaps in some cases this would be enough, but recall the whole data partition is restored - you can't choose individual files or apps. In my case this corrupted some system settings but the data and apps were still there. I used Titanium backup to back those up to external storage, then flashed firmware and ROM again.
Whenever the phone is stuck use the "hard shutdown method" by holding the volume up and power button and waiting for the phone to vibrate 3 times. At this point the phone is off.
The first thing to do is get into recovery mode with TWRP.
(If you already can do this, skip to step 3.)
Notes About Sony Devices:
The Xperia devices have a recovery-in-boot arrangement. This means that the recovery is booted using the regular kernel / boot image in the device. Team Win has worked with the FreeXperia device maintainers to come up with a way to extract the ramdisk from the FOTAKernel partition and use the ramdisk from that partition instead of the recovery that is included in the boot image of your device. This means that if you install current CM nightlies and flash TWRP to the FOTAKernel partition, you will be able to use TWRP instead of the CWM or CM recovery that normally comes in a CM boot image. Other boot images including stock kernels can be repacked to include this extraction utility to allow you to use TWRP from the FOTAKernel partition. This setup allows you to choose what recovery you want to have installed and allows you to update your recovery more easily. Unfortunately this setup requires that the boot image that you have installed include the ramdisk extraction utility.
TWRP may not work on your device unless you install other custom software as well.
You will need adb and fastboot, in Windows represented as
fastboot.exe which run in command line terminal - (for adb and fastboot Windows XP click here)
If you do not have TWRP, follow these instructions (for the Xperia XZ1 Compact)
a) download the ZIP file from here (it is that TWRP but with working adb)
b) extract the img from the ZIP file (not a flashable zip)
c) connect the phone to the computer via a USB cable in fastboot mode. To put the phone in fastboot mode, completely power it off. Then hold the volume up button and the power button at the same time until the light on the phone turns on. connect the phone to the USB cable if it's not already.
d) If you're on Windows, open a command prompt with administrative privileges by right clicking on it and choosing "run as administrator". I recommend to use
cmd.exe and not the Windows PowerShell. Navigate to the
platform-tools folder with the
cd command. Type in
fastboot devices. If nothing is returned or it says "unauthorized" that means you need to install the correct drivers. The official drivers are here but I found I couldn't install them directly. I needed to download the Google USB Driver and replace
android_winusb.inf with that from Sony. I found I needed to reboot using advanced options to disable Windows Driver Verification in order to install the drivers. Right click on the
inf and choose "install".
Now that the drivers are installed, from the elevated command prompt run
fastboot flash recovery C:\path\to\folder\twrp-3.3.1-0-lilac-android10-2.img
Boot to recovery. This can be done by turning off the phone, holding the volume down button and power on button at the same time until the screen turns on.
(If it doesn't work you can also boot image from fastboot without flashing)
fastboot boot C:\path\to\folder\twrp-3.3.1-0-lilac-android10-2.img
If your phone uses FBE file-based encryption the data will be encrypted with lock screen password/PIN (hence the strange file names in
/data/data such as
+NiZZaTrs3RFzYegpkEk). Newer versions of TWRP support decryption. TWRP should prompt you for a password when it first starts, however I found this buggy and it sometimes just didn't ask for a password. If this happens try restarting the phone in recovery or reflashing TWRP. Also going into terminal emulator from Advanced, try the command
twrp decypt [password], see here if you had a pattern.
Backup your (decrypted) data from TWRP backup menu. make sure backup location is MicroSD Card or USB storage. Only choose the Internal Storage if you planning to save all your files to PC before factory reset.
Don't forget to backup Internal Storage manually as it is not included in TWRP backup. You can do this via cmd line
adb pull /sdcard C:\path\to\folder
Note for adb to work, the phone must be in recovery mode (i.e. TWRP is open)
Download Flashtool, a ROM of your choice and compatible firmware. You can try to flash the same ROM that you have been using, but I was unable to find stock ROM for this phone. I recommend Lineage 16 with this firmware (Lineage requires 47.2.A.11.228). I tried Lineage 17.1 but setup would go into a loop. Also download GApps Pico for Arm64 and Android 9. Download Magisk zip and apk. This is needed if you're going to use Titanium Backup to backup the apps and data you want to save. Perhaps in some situations this isn't needed if you can migrate without root.
In Flashtool flash the FTF for the firmware first. Then flash the ROM (e.g. Lineage).
Boot into TWRP and install Gapps pico and Magisk. Magisk is needed for root which is needed by Titanium Backup, which will be used later.
The phone should be able to boot normally now. Give it plenty of time (e.g. 10 minutes). Take a break to go for a walk.
Confirm everything is working relatively ok. For example browse the web, download a simple app from the play store. If it's not, flash a different firmware or ROM with Flashtool.
Install and setup Titanium Backup. If you prefer alternative you can also use Migrate! (from the Google Play Store or anywhere else)
Reboot to TWRP
Restore only the data partition.
Clear cache and reboot to system.
In my experience all the data and apps were restored. But some system functionality was broken (like the WiFi). If it wasn't we would be done on this step :)
Use Titanium Backup to selectively backup the apps and data you want to keep. Make sure the backup is put on an SD card or USB storage so it won't get over written.
If your device is not working, there is also a method to do this 'offline' from recovery. refer to the Migrate support thread on forum.xda-developers.com (development in progress)
Do a factory reset from TWRP -> Wipe -> Format Data -> type 'yes'
if Format Data fails, Repeat step 7 and 8.
The phone should be working normally again.
Use Titanium Backup or Migrate to restore the apps and data from step 18.