Since Android is based on Linux kernel, getting root access by executing /system/xbin/su - like one can do on a Linux system - used to work in good old days but not now. The story is a bit twisted.
WHAT IS SUID?
Special thing about /system/xbin/su isn't that it's in /system partition, but the set-user-ID-root (SUID) bit set on this binary makes it special. SUID (Set owner User ID up on execution) is a special type of permission on Linux kernel that grants the permissions of the file owner to the new process rather than inheriting permissions from parent process who runs it. Since the owner of /system/xbin/su is root (UID 0), it can switch the effective UID of process to 0 when executed, hence enabling calling process to have root access.
This is a part of Discretionary Access Control (DAC); one of Linux kernel's security features.
MANDATORY ACCESS CONTROL (MAC):
Implemented since KitKat, SELinux can control who executes su binary, by defining Access Vector rules.
On userdebug/eng build of a ROM, adb shell (by executing su, as you have mentioned) and adb daemon (1) can make transition (2, 3, 4, 5) to run as root with unrestricted context u:r:su:s0 (6). But adbd can't be run as root on final user builds (7, 8), neither su binary is included. So, /system/xbin/su is intended only for developers to debug engineering or userdebug builds of a ROM, not for end user (9). And second special thing about it is its SELinux context u:object_r:su_exec:s0.
See Why “adb root” does nothing? for more details.
HOW ANDROID RESTRICTS ROOT ACCESS?
Keeping SELinux aside, on user build of a ROM that is intended for end users, placing su under /system won't give any advantage, even with set-user-ID-root. A normal app can't elevate its privileges to root by executing su because:
- Starting with Jelly Bean, Android switched to file capabilities (a capability is a subset of root privileges) instead of relying on
set-user-ID type of security vulnerabilities. A more secure mechanism: Ambient capabilities has also been introduced in Android Oreo.
- System daemons and services can make use of file capabilities to gain process capabilities (see under Transformation of capabilities during execve) but apps can't do that either because they run with process control attribute
NO_NEW_PRIVS set, ignoring set-user-ID as well as file capabilities. SUID is also ignored by mounting /system and /data with nosuid option for all apps.
- UID can be switched only if calling process has SETUID/SETGID capability in its Bounding set (one of the 5 capabilities categories a process can have). But Android apps are made to run with all capabilities already dropped in all sets.
- Starting with Oreo, apps' ability to change UID/GID has been further suppressed by blocking certain syscalls using seccomp filters.
ADB daemon, however runs with CAP_SETUID and CAP_SETGID (to allow switching to UID/GID shell???). But adb shell too can't get us capabilities by executing a SUID-root su binary because NOROOT and NO_SETUID_FIXUP security bits are set on adbd process. Contrarily on a debugging build, no capabilities are dropped from bounding set (10), so executing /system/xbin/su (with set-user-ID-root bit or cap_setuid,cap_setgid+ep file capabilities) would get us full root access.
Syscalls used in these sandboxing methods: cap_set_proc, setresuid, setresgid, sys_seccomp, prctl (CAPBSET_DROP, SET_NO_NEW_PRIVS, SET_SECUREBITS, SET_SECCOMP) are all part of Minijail.
Summarizing the above lines, DAC and MAC would only allow adb root and executing /system/xbin/su from adb shell on userdebug build of ROM. To have root access from apps and on user builds targeted for end user, we need to root the phone as explained below.
HOW TO GET ROOT ACCESS ON ANDROID?
So if we want to have privileges with UID 0 and all 38 capabilities of Linux kernel, we need a rooting solution that can deal with all above described security measures. Since the standalone su binaries stopped working with the release of Jelly Bean, a transition was made to su daemon mode.
Now on a rooted phone, a fully privileged daemon runs from the very start of booting process (in fact init is replaced). When an app needs root access, it executes su binary provided by the rooting solution. This su doesn't change UID/GID on its own, but just connects to super user daemon through a UNIX socket and asks to provide the requesting app a root shell with all capabilities. In order to interact with user to grant/deny su requests from apps, su daemon is hooked with an app that can display user interface prompts. A database of granted/denied permissions is built by su daemon for future use.
In addition to that, SELinux denials are also handled by defining SELinux policy allow-rules when rooting the phone.
New methods of root like Magisk work in system-less mode i.e. without modifying /system partition. Only boot partition - that contains kernel and initramfs - is modified to inject su daemon service and new SELiux policies. A locked bootloader won't boot modified boot.img, so bootloader needs to be unlocked. For more details regarding this, see Unlocking Bootloader. However there have been some rooting hacks that worked by bypassing above described security implementations without going for proper way of rooting. Most of such exploits and vulnerabilities in Android OS have been fixed over time. Here is a good write up on this subject.
So to conclude, placing su binary under /system or any other location won't give any benefit.
Also see How Magisk works? for more details.
PS: Please ignore so many links, they are for my own future reference.
