How to install a firmware without losing your data and restore the previous one in case it goes wrong?

Question

I have a OnePlus 5T and I can't install OTA updates anymore because my firmware is too old. In addition to that I don't have a full backup of my device because I'm using encryption and TWRP doesn't understand it.

I would like to simply try the new firmware and revert in case anyhting goes wrong without losing any data. Is that possible?

I'm not exactly sure what firmware means in this context and where to get it, but looking at the website, there is this download for a zip file with the following contents:

I'm pretty sure that this is the complete stock ROM, meaning if I were to install it, it would wipe everything and replace my LineageOS with the stock ROM.

What I would like to do is to simply flash "the firmware" and only the firmware without losing data on my internal storage and also not my app data, contacts, sms, wifi passwords, bluetooth pairings etc etc.

And before that I would like to create a backup of my old firmware so that I can go back in case it goes wrong.

Can this be done? Can it be done with fastboot?

Maybe using fastboot flash PARTITION [FILENAME] and some other command to do the backup?

Edit:

The updater-script in META-INF/com/google/android/update-script contains the following code:

getprop("ro.display.series") == "OnePlus 5T" || abort("E3004: This package is for \"OnePlus 5T\" devices; this is a \"" + getprop("ro.display.series") + "\".");
is_part_existed("/dev/block/bootdevice/by-name/vendor") || abort("vendor partition is not existed, exit ota!!");
show_progress(0.650000, 0);
ui_print("Patching system image unconditionally...");
block_image_update("/dev/block/bootdevice/by-name/system", package_extract_file("system.transfer.list"), "system.new.dat", "system.patch.dat") ||
  abort("E1001: Failed to update system image.");
show_progress(0.100000, 0);
ui_print("Patching vendor image unconditionally...");
block_image_update("/dev/block/bootdevice/by-name/vendor", package_extract_file("vendor.transfer.list"), "vendor.new.dat", "vendor.patch.dat") ||
  abort("E2001: Failed to update vendor image.");
show_progress(0.050000, 10);
show_progress(0.050000, 5);
package_extract_file("boot.img", "/dev/block/bootdevice/by-name/boot");
show_progress(0.200000, 10);
ui_print("Writing static_nvbk image...");
package_extract_file("RADIO/static_nvbk.bin", "/dev/block/bootdevice/by-name/oem_stanvbk");

# ---- radio update tasks ----

ui_print("Patching firmware images...");
ifelse(msm.boot_update("main"), (
package_extract_file("firmware-update/cmnlib64.mbn", "/dev/block/bootdevice/by-name/cmnlib64");
package_extract_file("firmware-update/cmnlib.mbn", "/dev/block/bootdevice/by-name/cmnlib");
package_extract_file("firmware-update/hyp.mbn", "/dev/block/bootdevice/by-name/hyp");
package_extract_file("firmware-update/pmic.elf", "/dev/block/bootdevice/by-name/pmic");
package_extract_file("firmware-update/tz.mbn", "/dev/block/bootdevice/by-name/tz");
package_extract_file("firmware-update/abl.elf", "/dev/block/bootdevice/by-name/abl");
package_extract_file("firmware-update/devcfg.mbn", "/dev/block/bootdevice/by-name/devcfg");
package_extract_file("firmware-update/keymaster.mbn", "/dev/block/bootdevice/by-name/keymaster");
package_extract_file("firmware-update/xbl.elf", "/dev/block/bootdevice/by-name/xbl");
package_extract_file("firmware-update/rpm.mbn", "/dev/block/bootdevice/by-name/rpm");
), "");
ifelse(msm.boot_update("backup"), (
package_extract_file("firmware-update/cmnlib64.mbn", "/dev/block/bootdevice/by-name/cmnlib64bak");
package_extract_file("firmware-update/cmnlib.mbn", "/dev/block/bootdevice/by-name/cmnlibbak");
package_extract_file("firmware-update/hyp.mbn", "/dev/block/bootdevice/by-name/hypbak");
package_extract_file("firmware-update/tz.mbn", "/dev/block/bootdevice/by-name/tzbak");
package_extract_file("firmware-update/abl.elf", "/dev/block/bootdevice/by-name/ablbak");
package_extract_file("firmware-update/keymaster.mbn", "/dev/block/bootdevice/by-name/keymasterbak");
package_extract_file("firmware-update/xbl.elf", "/dev/block/bootdevice/by-name/xblbak");
package_extract_file("firmware-update/rpm.mbn", "/dev/block/bootdevice/by-name/rpmbak");
), "");
msm.boot_update("finalize");
package_extract_file("firmware-update/logo.bin", "/dev/block/bootdevice/by-name/LOGO");
package_extract_file("firmware-update/NON-HLOS.bin", "/dev/block/bootdevice/by-name/modem");
package_extract_file("firmware-update/adspso.bin", "/dev/block/bootdevice/by-name/dsp");
package_extract_file("firmware-update/BTFM.bin", "/dev/block/bootdevice/by-name/bluetooth");
set_progress(1.000000);

Should I just take the msm.boot_update("main"), and the msm.boot_update("finalize"), part and flash them like this?

# main
fastboot flash cmnlib64 ./firmware-update/cmnlib64.mbn
fastboot flash cmnlib ./firmware-update/cmnlib.mbn
fastboot flash hyp ./firmware-update/hyp.mbn
fastboot flash pmic ./firmware-update/pmic.mbn
fastboot flash tz ./firmware-update/tz.mbn
fastboot flash abl ./firmware-update/abl.elf
fastboot flash keymaster ./firmware-update/keymaster.mbn
fastboot flash xbl ./firmware-update/xbl.elf
fastboot flash rpm ./firmware-update/rpm.mbn

# finalize
fastboot flash LOGO ./firmware-update/logo.bin
fastboot flash modem ./firmware-update/NON-HLOS.bin
fastboot flash dsp ./firmware-update/adspso.bin
fastboot flash bluetooth ./firmware-update/BTFM.bin

Or would this already overwrite a partition that contains some sort of user data? Can I make a backup of these partitions first? And what about RADIO/static_nvbk.bin, boot.bin, system.transfer.list, vendor.transfer.list and all that msm.boot_update("backup"), stuff?

Edit: I ended up not being able to flash those firmware files via fastboot because it requires more than a "normal" unlocked bootloader. If I had done the necessary step to unlock the bootloader further it would have wiped all my data. Fortunately you can bypass that by creating a zip with the firmware files and a custom OpenRecoveryScript (updater-script). I found an awesome GitHub project that automates the whole process: https://github.com/angela-d/firmware_oneplus

I simply sideloaded the generated zip via adb sideload firmware-update-oneplus5T.zip and finally I was able to install my OTA updates.

https://github.com/angela-d/firmware_oneplus

Answer 1

1) You can boot into TWRP recovery from fastboot

fastboot boot twrp.img

2) Then you can backup your whole emmc from adb

adb pull /dev/block/mmcblk0

Does /dev/block/mmcblk0 include my internal storage, my app data, contacts, sms, wifi passwords, bluetooth pairings, system settings etc? Have you personally tried that method on an encrypted device?

It is all stored in userdata partition. If TWRP is able to decrypt, you should pull /dev/block/dm-0 (=userdata) to get the unencrypted backup.

Otherwise, you still can separate encrypted USERDATA partition (and correlated partition EFS/METADATA required for decryption) from mmcblk0

Do a research how encryption works for your device, i can't help with which partitions are required!

3) print partition table with start/size

parted mmcblk0 unit B print

Warning: Not all of the space available to mmcblk0 appears to be used, you can fix the GPT to use all of the space (an extra 991 blocks) or continue with the current setting? 
Fix/Ignore? i                                                             
Model:  (file)
Disk mmcblk0: 15300820992
Sector size (logical/physical): 512/512
Partition Table: gpt
Disk Flags: 

Number  Start         End           Size          File system  Name         Flags
 1      524288        3670015       3145728                    proinfo      msftdata
 2      3670016       8912895       5242880                    nvram        msftdata
 3      8912896       19398655      10485760      ext4         protect1     msftdata
 4      19398656      29884415      10485760      ext4         protect2     msftdata
 5      29884416      30146559      262144                     seccfg       msftdata
 6      30146560      30539775      393216                     lk           msftdata
 7      30539776      47316991      16777216                   boot         msftdata
 8      47316992      64094207      16777216                   recovery     msftdata
 9      64094208      64618495      524288                     para         msftdata
10      64618496      73007103      8388608                    logo         msftdata
11      73007104      83492863      10485760                   expdb        msftdata
12      83492864      84541439      1048576                    frp          msftdata
13      84541440      118095871     33554432      ext4         nvdata       msftdata
14      118095872     159383551     41287680                   metadata     msftdata
15      159383552     161480703     2097152                    oemkeystore  msftdata
16      161480704     167772159     6291456                    secro        msftdata
17      167772160     176160767     8388608                    keystore     msftdata
18      176160768     2801795071    2625634304    ext4         system       msftdata
19      2801795072    3070230527    268435456     ext4         cache        msftdata
20      3070230528    15283519487   12213288960   ext4         userdata     msftdata
21      15283519488   15300296703   16777216                   flashinfo    msftdata

4) copy the partitions into single files (partitions vary for each device model, this is just example for FDE. sometimes metadata is a file located at EFS partition, sometimes encryption footer is concatenated to userdata partition itself)

dd if=mmcblk0 of=metadata.bin skip=118095872 count=41287680 bs=1
dd if=mmcblk0 of=userdata.img skip=3070230528 count=12213288960 bs=1

this is just example for better understanding and is very slow (10 kB/s). of course dd will copy faster when we increase block size, so calculate bytes into 8k blocks

12213288960 / 8192 = 1490880

dd if=mmcblk0 of=metadata.bin skip=14416 count=5040 bs=8k
dd if=mmcblk0 of=userdata.img skip=374784 count=1490880 bs=8k

5) repeat this for all required partitions. for successful decryption system and vendor may required too. beware you can only decrypt with correlating rom on this unique device itself. decryption on other device (even same model won't work)

6) for restoring you can flash this partition dumps from fastboot

fastboot flash userdata userdata.img
fastboot flash metadata metadata.bin

another method is restoring from adb

adb push userdata.img /dev/block/bootdevice/by-name/userdata
adb push metadata.bin /dev/block/bootdevice/by-name/metadata

or even from within twrp terminal

dd if=/external_sd/userdata.img of=/dev/block/bootdevice/by-name/userdata
dd if=/external_sd/metadata.bin of=/dev/block/bootdevice/by-name/metadata

FIRMWARE UPDATE

for firmware update i recommend to flash complete stock rom (you will lose all data), then flash LineageOS, then restore twrp backup data.ext4.win* (which of course you have created from working TWRP before)

another method is using sdat2img to convert system.new.dat and vendor.new.dat into ext4 images system.img and vendor.img, which you can flash along with other partitions from fastboot

please also read the questions from Android Enthusiasts Linked and Related section