I have a Nexus 5 with TWRP as my recovery.
I've encountered and issue over the past few months.
When I reboot to recovery from the OS, or from Fastboot, it'll stay at the boot screen a bit longer than usual, but then boot the OS as if I didn't try to get to recovery. I can usually expect this 2-3 times and then it'll boot into recovery correctly like nothing happened.
It's been across multiple versions of TWRP and I've reflashed the current one a few times to ensure it wasn't something along those lines.
Booting to fastboot always works, it's just booting to recovery where I see this.