/boot, /system, /vendor, /vbmeta and ODM Partitions are protected by android verified boot (AVB). They are not encrypted but their integrity is verified during boot. Any modification to these partitions will halt the boot flow and brick the device. At this point without EDL mode, you won't be able to flash stock OS to unbrick it.
AVB verifies vbmeta partition image using OEM public key which is hardcoded in android bootloader (ABL). Then hash of boot image is calculated and compared with the hash stored in vbmeta. Once the boot image is verified, kernel constructs hashtree of every partition and compares their root hash with the ones that are stored in vbmeta. In this way, just by protecting vbmeta, every other partition can be verified on boot.
To protect ABL from tampering (e.g. replacing OEM public key with your own and resigning vbmeta with your private key, thus breaking chain of trust), chipmakers implement secure boot. In Qcom devices, Primary Bootloader (PBL) which is burned on CPU die, verifies Xtended Bootloader (XBL) using Qcom's public key which is stored in eFuse. Then Xtended Bootloader verifies ABL and ABL enforces AVB.
When you unlock bootloader (unlock and unlock_critical), AVB is not enforced but secure boot is still enforced by SoC. This chain of trust goes upto the hardware level making any modification useless. There's no reason to encrypt system partitions as those images are already public. What matters is protection of their integrity.