Title says most of it. I use ES File Explorer, but the issue seems to exist in any file manager. Copying files to the SFTP server nets 20-30MB/s, whereas copying files from is ~400-500kB/s.
I gather this may have something to do with Android treating these copy operations as a background download, but that's mostly supposition.