Newer Android devices don't support USB mass storage because it has a big shortcoming: the phone and the PC can't access the storage at the same time that way. This is because USB mass storage is a low-level protocol giving the PC low-level access to the whole filesystem. This causes knock-on problems when you connect your phone to the PC:-
- Apps on the external storage can't run while it is mounted on the PC.
- Music and other files on the external storage aren't accessible to apps, so even apps that aren't on the external storage might stop working or become useless.
- When the external storage is unmounted from the PC, Android has to rescan it from scratch (see media-scanner) because it can't tell what has changed.
- There's no way for the phone to let the PC access the internal storage this way: Android can't unmount the internal storage, because all apps need access to it, even the system apps.
- This also means there has to be a separate partition and filesystem for the external storage, which the phone can cope with losing access to. Newer Android devices often just have one big partition, and the "external storage" is simply a directory there. It's more convenient that way because you don't have free space split between partitions, but you can't use USB mass storage on only a directory.
- Android can't enforce any filesystem security. The PC has complete access to the exported storage, below the filesystem level. This lets it access any file, and read the raw device data to recover deleted files. This might be considered an undesirable security hole.
- As eldarerathis points out, the external storage has to use the FAT32 filesystem to work this way. This isn't a shortcoming when it's a real SD card, but when it's really non-removable storage, it would be more useful to use a better filesystem.
(Where I say "SD card" here, that means whatever storage your Android device calls "external", even if it's not actually removable.)
In contrast, MTP is higher-level, and allows Android to arbitrate and to share access, just like when sharing directories or filesystems over a network. It even opens the possibility of letting the PC access the internal storage this way, but there are other restrictions around that unrelated to MTP.
As eldarerathis explains, you need special software to access MTP devices from a Mac, because Mac OS doesn't have the appropriate software built-in, like Windows and some Linux distributions do.