From experience
Does Google Play Protect only scan apps installed via the Google Play Store or does it scan all installed apps?
It checks for all apps installed on the device irrespective of source (barring systemless installs using Magisk). For instance V4A (Viper for Android) flashed through TWRP triggered this warning
Does it scan APK files on the device that are not installed?
No, it does not scan APKs. I was transferring APKs of old versions of apps which I prefer over current version and inadvertently copied Lucky Patcher APK. Since I didn't install it, it passed checks
