In the custom recovery TWRP, what does the option "zip file signature verification" do? How do I know if it should be checked when installing something?
Basically the Zip File Signature Verification flag will enable flashing only if the zip file is signed properly by the developer. This is (almost) the same method used signing Jar files in Java.
Basically, before executing any third-party program you want to make sure that it hasn't been tampered with (integrity) and that it was actually created by the entity that it claims to come from (authenticity). Those features are usually implemented by some digital signature scheme, which guarantees that only the entity owning the signing key can produce a valid code signature. The signature verification process verifies both that the code has not been tampered with and that the signature was produced with the expected key.
Note from the above, that this (obviously) cannot detect if the code itself is malware etc, just that it is as it was when it was signed by the developer. You have to trust the developer of any software you flash.
An example of this is that you cannot flash a custom ROM through a Stock Recovery as the stock recovery will look for a signature from the manufacturer.
It can also detect if a zip is corrupt but is not a definitive check, you would be better verifying the MD5 sum of the file and comparing to the one provided by the ROM developer.
"How do I know if it should be checked?" This is probably going to vary depending on what you are doing, I generally have it left at the default value of the particular recovery I'm using, and I've never actually had to check or uncheck it for any Zip files. Keep in mind some perfectly functional packages might be signed incorrectly and may install perfectly if you disable the verification, but conversely it could be a corrupt file and bootloop the device.
I leave it checked, and if someone on a thread / the developer says it's ok to install then I will make a nandroid backup and try flashing it with the signature verification turned off. (ALWAYS make a backup!)
Here is an example of trying to install a custom ROM on a stock S4 Recovery:
Finding update package...
Opening update package...
Verifying update package...
E:No signature (188 files)
Here's a link to the more technical information on GitHub which is a bit out of scope to go into on here.