Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
You don't need to "acept all connections". Just make sure to backup the adb authentication keys from your PC once you have accepted them on device. Then nothing can go wrong if you ever need adb access.
adb
supports RSA key authentication (if configured) which is far stronger than a password: android.stackexchange.com/a/219320/218526