The "proper" answer would be exFAT. "Proper" because exFAT is crap but since it does "fix" some of FAT's problems, it is the only hassle-free option.
Ext4 would be a much better option. Should it be properly managed in Android regarding rights management - from my quite recent understanding (latest CM11 as of Jan 2015), Android does not expect the true external storage to understarnd, let alone enfore access rights. Since *FAT has no concept of any rights, this is no problem. However, in Linux there is no clean way of disabling rights management on a native filesystem. Android's concept of security uses UNIX access rights in a different way than it was intended. Not to blame Android, the UNIX concept is ancient but it's use comes at a price. The price being the need of external rights management which Android is not ready for and the User has (usually) no control of either.
There is a way to use ext4 on Android, called Mounts2SD. It's quite clean and reliable though it is anything but native (made of "nasty hacks" some would say). I wouldn't recommend it unless you have absolute confidence in what you are doing.