I'm looking for a way to install a flashable zip (no OTA zip), using the "Install Update from SD card" option built into any stock recovery, without installing a custom recovery. Is there any way to do this? Will "unlocking OEM" in developer settings allow this?
Unfortunately this is not possible on devices newer than Android 5.0.
After 5.0, signature verification was enhanced to combat the Universal Signature bug.
Prior to Android 5.0, a bug existed in the Signature verification program, something to do with the way the signature was validated allowed another signature to result in the same validation ( The Universal Signature ) .
To answer your question for Android 5.0 and lower..
Repackage the custom system.img into an OTA Update format
Sign the update package with the Universal Signature
Hope that it validates through the stock program.
In most cases, the recovery program checks the format and signature are correct, then it check's if the package is designed for your device by reading the build.prop.
If your build.prop matches the package, and the signature is validated it will install.