I've been told that the SD card is a terrible place to store secure data because any app can read it. How else can my phone store my private key I use to log into my home PC ssh server?
I use ssh in a terminal.
So the private key is protected simply by the unix filesystem and Android's user-per-app design: only Better Terminal Emulator Pro can read id_rsa, and only ssh and other programs I'm aware of run under it.
Whatever your app is, it need only do this to enjoy the same security from other apps. Full-sdcard encryption, unless it comes packaged with a filesystem having a comparable system of permissions, will not protect your data from other apps.
If your phone is rooted, you have the option of placing the key in a 'protected' area in which any application requesting to access the file would need to be given root access by you.
You can also create a key that requires a pass-phrase to be entered in order to unlock and use the key.
As previously mentioned, Android is moving towards full disk encryption, which will substantially increase security of the device in general.