Running an SSH server on your phone helps you with this problem, just as the previous answer explained. However, because DroidSSHd is older and may not work with newer phones on more recent versions of Android, SSHelper might be a better choice, and I use it regularly for purposes similar to this.
Simple answer : NO
Hardware drivers are proprietary
Building them without the source is extremely difficult
Requires a lot of expertise and time
Not possible unless you have low level access to your device hardware (Meaning being able to flash without fastboot or edl)
So yeah stick to the answer that, NO you can't replace Android entirely with Linux....