The combination of encryption, PIN lock, and having ADB turned off should protect your data pretty well. It could still be compromised if someone finds a security flaw that can be exploited through the network or the MTP USB connection, but that risk exists in any case.
However, with your bootloader unlocked, someone could take your phone, flash a malicious ROM that contains keystroke loggers or something, and then return the phone to you and wait for you to type your PIN or decryption password. It'd be better to keep the bootloader locked whenever you don't actually need to flash things via fastboot.
You can use BootUnlocker to lock and unlock the bootloader from within the running Android system. Unlike fastboot oem unlock, this doesn't wipe your data when you unlock.