8

Is there a way how to offline safely synchronize directories between sd card on Android which is not rooted and PC with Ubuntu?

Safely = using secured connection between the devices (for example using ssh).

Offline = without Internet access, but the PC and smartphone are connected through LAN.

Synchronize directories = a process which transfers data between smartphone and PC in a way that at the end of the process, both devices contain exactly same data in selected directories (preferably, but not necessarily, excluding filtered files). The process is able to detect deleted data so it doesn't copy file which was originally on both devices and later deleted on one of them.

My systems

  • Android 6.0.1 (not rooted)
  • Xubuntu 18.04 64 bit PC

What I already tried/checked (without success)

Unison

Unison for Android

  • https://play.google.com/store/apps/details?id=com.nerdysoftware.unison
  • Not secure - from application description: "No SSH access is necessary - clients connect directly via socket. Because Unison clients connect to your device over a direct socket connection instead of over SSH, the server should only be run when your Android device is connected to a trusted network, like a home WiFi. Running the server on other networks could expose files on your device to untrusted parties."
  • Also seems to use older version of Unison (this application requires exactly same versions on server and client to allow data transfer).

Rsync

SSHDroid

  • https://play.google.com/store/apps/details?id=berserker.android.apps.sshdroid
  • For using SSH with another software (see bellow)
  • Cannot write/modify SD card using the ssh connection. I went to settings -> Applications -> SSHDroid -> Permissions: storage is enabled. There I opened context menu -> all permissions: reading SD card available. Modification/deleting data on SD card available. In other words, I checked the application has permission to modify the SD card: it has.

SSHFS with SSHHelper

SSHFS with SimpleSSHD

What I already tried (with mixed/partial success)

Combination: Unison (on PC) + some SSH server (Android) + SSHFS (on PC)

  • In case I could find working SSH server; use SSHFS to mount remote Android device locally and then use Unison for synchronization.
  • Results:
    • When the SSH is actually rw, then it works, but it's so slow that it's impractical - it must ask for every single file and directory over the network. Of course, other apps (like Unison) must do that too, but they don't ask for every single file sequentially (not: client send request for single file - latency - server checks and sends response - client receive response and go to next file).
    • Unison requires permission to rename files; it always creates temporary files and directories and when everything is downloaded and double-checked, then old files (if any) are deleted and new files renamed. So if renaming is not allowed, it doesn't work.

SSH/SFTP Server - Terminal

  • https://play.google.com/store/apps/details?id=net.xnano.android.sshserver
  • This actually works but has two problems;
    • It requires entering password several (>10) times when mounted as sshfs (I tried trivial one and and made sure it's not typo) and doesn't support ssh keys. I was able to solve this problem with a hack: zenity --password | sshfs user@host:/ "$PWD/data" -o reconnect,password_stdin, but I'm not sure how secure it is.
    • This ssh is unbelievably slow. Much slower when compared with SSHDroid and SSHHelper (not only with sshfs, but also when I simply connect using ssh and terminal; typing and responses are really, really slow). Maximum speed is 2 MB/s, but most of the time it's ~500kB/s.
    • Cannot rename files (permission error) so it makes it unusable with Unison (on PC).

Syncthing

  • https://play.google.com/store/apps/details?id=com.nutomic.syncthingandroid
  • Problem: cannot modify SD card (on non-rooted systems) and devs are not going to fix this anytime soon.
  • The only solution is to use obscure, hard-to-reach deep directory structure: /storage/XXXX-XXXX/Android/data/com.nutomic.syncthingandroid/files.
  • But for this particular directory, it works. It's speed is about 5 - 10 MB/s and I suspect that it's being throttled down by SD card itself (or Android). For now, I'm using this solution, but the directory is really inconvenient.
  • Smaller problems:
    • It doesn't support following symlinks and until someone (other than current devs) implement it, it's not going to be supported.

What problem I'm trying to solve

I "just" want to synchronize data on file level (some pdfs, images, etc.) between my PC and Android smartphone. I use multiple devices (desktop, laptop, etc.) and use Unison (note: I do not insist on this app) to synchronize data between them. Because I switch environment quite offten, I need it to work on one button press and to be fast as possible (which, for example, Unison does greatly). Because I work on different environments, I need to cover my cases: sometimes I'm without Internet access. Other times I'm connected to untrusted network.

This is also reason why I cannot use cloud (like Dropbox): it's slower than two devices communicating directly and sometimes I'm without Internet connection so I cannot synchronize even when the devices are connected on same network.

The reason why I want to sync with SD card is simply because it has bigger capacity so my files fits.

Improve this question
12
  • 1
    Upvoted for thorough spade work :) Welcome to our site
    – beeshyams
    Nov 3, 2018 at 18:48
  • 1
    Not tried, but maybe installing Nextcloud on your Linux machine and the Nextcloud app on your Android device? Nextcloud comes with a sync client AFAIK. Alternatively, if Playstore is acceptable, FolderSync works via SFTP (ie. SSH) and wouldn't require additional software on your Linux machine.
    – Izzy ♦
    Nov 3, 2018 at 20:23
  • 1
    No, it's not the fastest – but that never bothered me. It syncs in bg at 3am while I sleep, and is done when I get up. Conflicts are the real bad thing here indeed – which I usually notice when having copied an entire folder to a new device (FS seems unable to detect their state and doesn't even seem to bother trying CRC or something else to find out). But unfortunately, no better ideas here. Be welcome to check my corresponding app listing, there are many candidates I didn't try myself ;)
    – Izzy ♦
    Nov 4, 2018 at 16:01
  • 1
    @Izzy I guess it depends on our workflows - I usually do some work to the last minute, then sync, take my things and run to catch a train. (About FS - yeah, it seems to me exactly as you wrote.) Your list of apps is really interesting, thanks for sharing. I'll try to find some time to check out some apps from it :).
    – Tom
    Nov 4, 2018 at 16:11
  • 1
    @Izzy Well, it seems good, except it doesn't work on SD card; see links in my question (I mention the app there).
    – Tom
    Nov 4, 2018 at 17:09

2 Answers 2

Reset to default
3

At the end, I settled with Syncthing. So, how does it fulfil my requirements:

Safely

It uses encryption. I didn't analyse how secure it is, so this is more matter of trust, but at least it shouldn't send data unencrypted.

Offline

Works exceptionally well. Local LAN detection seems reliable and is able to connect clients directly. I use it also at work (Windows-to-Windows and Windows-to-Linux) to share data with co-workers without exposing them to 3rd-party companies, it's fast and reliable. (Tens of gigabytes are transferred pretty quickly, so it seems the only bottleneck is the local network.)

Synchronize directories

Works well, but at the end, I decided to synchronize two-ways only one subdirectory instead of all data. It seems like Android likes to modify my files without my consent (especially media files), so except certain working directory, I block any changes done by the Android. It's inconvenient, but it's not issue with Syncthing.

Another problem is when synchronizing big files (thousands of megabytes and more), but again, it seems like issue with my device, but after a few system crashes, Syncthing eventually manages to transfer the data. (Later edit: it seems this issue really happens only on my Android device. Syncing single ~25GB file with a co-worked, Windows-to-Windows was without any problems.)

Pros

  • Reliable across platforms.
  • You just configure it and it works in background. If you're used, for example, to Dropbox, it's similar, except you don't share your data with 3rd party company. And it's actually faster (ymmv), because it synchronizes data between clients directly, not with server.
  • The application can be configured through browser. You can enable access from outer device (I recommend to enable password in this case!!!) which allows you to configure Android Syncthing through your desktop browser (you just need to find out IP address of your phone), which is really convenient.

Cons

  • As already mentioned in question, only directory on SD card, which can be synced, is /storage/XXXX-XXXX/Android/data/com.nutomic.syncthingandroid/files. It's really inconvenient, but I got used to it, somehow.
  • Linux desktop version doesn't handle symlinks. I solved it by using mount bindfs --resolve-symlinks --no-allow-other "srcDir" "shareDir".

Notes

I miss option to review data changes before they're synchronized, but this is matter of preference and application design, not an issue. And as stated in question, I couldn't find reliable Unison implementation on Android.

Improve this answer
6
  • I also chose Syncthing so far, have you experienced sometimes files getting replaced by 0 byte ones as well? I'm trying to figure out if Syncthing is to blame or whether something else is wrong with my phone...
    – Tobias Kienzler
    Jan 8, 2022 at 8:01
  • 1
    @TobiasKienzler I use Syncthing since the day I wrote the answer. On one side, no, I didn't experience files being replaced by zero bytes. However, on the other side, I sometimes experience that old files are damaged (i.e. just some bytes are changed, I didn't check which ones exactly.) Personally, I suspect the problem is old memory card, but I never inspected it to more depth (I only created a git repository to track what's happening - the repo itself got damaged after more than a year.) I just sync important data one way PC -> phone and move working data (e.g. photos) to the safe directory.
    – Tom
    Jan 22, 2022 at 10:21
  • 1
    @TobiasKienzler Also, I sometimes use Syncthing for sharing big data (tens of gigabytes) with a colleague. That's PC -> PC connection (no phone involved) and I never experienced any such issue, so I'm convinced the problem lies in the phone (although we use it much, much less frequently than how I synchronize my phone, so it's not exactly same case.) (And btw. sorry for late answer.)
    – Tom
    Jan 22, 2022 at 10:27
  • 1
    @TobiasKienzler Well if it also happens on the internal storage, it's quite suspicious. In that case, I would recommend to split the data into two parts - one synced through Syncthing, the other manually (e.g. through ssh) and watch them thoroughly and see what happens.
    – Tom
    Jan 23, 2022 at 12:55
  • 1
    Indeed, I guess I'll have to check it in more detail... Thanks for your feedback!
    – Tobias Kienzler
    Jan 24, 2022 at 7:32
0

I used to plug in the mobile and mount the directories using simple-mtpfs

  mkdir /mnt/phone
  simple-mtpfs -o enable-move /mnt/phone

I could then use unison directly between appropriate directories

It worked great for many years over 3 different phones.

Additional I was hoping to use unison for android, but it cant create a lock file on the sdcard which it wants to use for its one phone files. (Mistakenly thinks there is one, and asks to delete it)

Only problem is the plug on my current phone is so loose I can't get a file transfer connection, only power charging. Which is why I am here looking for a network solution.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .