It has to do with dm-verity. Taken from XDA Developers here:
The problem (it’s a problem if you like root and modifying devices)
stems from something I pointed out a long time back, when it first hit
AOSP – the introduction of dm-verity to Android. Verity is a security
feature, originally found in ChromeOS, designed to provide assured and
trustworthy computing devices, preventing malicious software from
modifying a device. Back in Android 4.4, Google announced verity for
Android, and then all remained quiet. While there has been some
research into using verity, for the most part, things have been quiet.
Until now, that is.
With Android 6.0, Google has begun to up their game on device
security. One of the fundamental requirements for this is to prevent
the software on a device from being modified without a user’s
knowledge – while many here at XDA take root for granted, imagine a
user’s device being rooted without their knowledge or consent, and
root access being used to steal their data. For this reason, Google
has started to implement verification of the system partition on some
To answer your question of "How does it know..."...any changes to the system partition at boot (or when accessed) could cause one of three warning/error messages.