Why do write attempts to /system/priv-app cause /system to switch to read-only?

Question

I'm trying to write files to the /system/priv-app directory. I boot into recovery (TWRP) and run adb shell.

~ # mount /system
~ # mount
rootfs on / type rootfs (rw,seclabel)
...
/dev/block/mmcblkXXXX on /system type ext4 (rw,seclabel,relatime,data=ordered)

I can then run:

~ # mkdir /system/tmp
~ # touch /system/tmp/test

This works fine: I have no problem modifying /system. (I can also delete things from /system/priv-app.) But if I then try to write to /system/priv-app I get the following:

~ # mkdir /system/priv-app/test
mkdir: can't create directory '/system/priv-app/test': Read-only file system
~ # mount
rootfs on / type rootfs (rw,seclabel)
...
/dev/block/mmcblkXXXX on /system type ext4 (ro,seclabel,relatime,data=ordered)

Now /system is suddenly mounted read-only! Permissions and ownership are identical in /system and /system/priv-app and I'm running as root.

~ # ls -al /
...
drwxr-xr-x   18 root     root          4096 Nov 25 07:32 system
~ # ls -al /system
...
drwxr-xr-x   59 root     root          4096 Aug  5  1972 priv-app

While in TWRP, SELinux is set to permissive:

~ # getenforce
Permissive

I'm running Lineage OS 14.1 (Android Nougat) on a Moto G 2015 (aka Moto G3). It's rooted with the official Lineage su addon. I also tried Magisk and get the same result.

I've run out of ideas, and I can't figure out the right search terms, because I've tried a ton of things and have not turned anything up.

How can I enable writing to /system/priv-app?

Answer 1

Protecting Android from malicious codes has always been a priority of Google, OEMs and SOC vendors. Even the /system write protection has been implemented on kernel level by some OEMs. And by default, Android mounts /system read-only in order to avoid any intentional or accidental changes for AVB/dm-verity to work properly. Only OTA updates can modify system partition.
However if dm-verity is disabled while rooting the device or when flashing custom kernel/ROM, we can mount /system read/write for modifications. But some programs have a built-in code to check if /system is mounted rw, and if so, revert the status to ro. Solid Explorer states:

If the mount point is mounted in r/o mode it will be temporarily remounted to r/w. After the operation completes (with a success or not), the app mounts the partition back to r/o state.

TWRP has also an option mount_sys_ro_chk and by default it mounts /system read-only.
So the solution is to change the mount point from /system to something else as that won't be checked usually.

Another factor that may force ro mounting of a partition is Linux kernel's mount option errors=remount-ro. This mount option is usually default on a newly created filesystem:

~# tune2fs -l /dev/block/bootdevice/by-name/system | grep Error
Errors behavior:          Remount read-only

So filesystem will be mounted as read-only if an error occurs e.g. Max Mount Count is reached and e2fsck isn't run. However this is rarely the case on Android. You can view kernel log for any such errors:

~# dmesg | grep mount

Changing the default behavior is not recommended and can be really harmful:

~# mkdir /system-temp
~# mount -o rw,errors=continue /dev/block/bootdevice/by-name/system /system-temp