2

I have an Android device that has no free space and no SD-card that I can replace (thanks to OnePlus policy). I also have no free space on PC to accommodate the backup so I wanted to upload it to the cloud. I have SSH and root access to the device. I also have a remote FTP server space (that I not trust, though).

I would like to make a password protected tar or zip file and upload it there. I'm not sure if a zip of / is sufficient for backup purposes.

The problem is there is no free space left on the device so I need to create the zip file on the fly and "stream" it to the backup server (it's SFTP on port 30).

I checked that the SSH, rsync and zip commands are available but I'm not sure how to do a full backup, encrypted zip compression and SFTP upload without writing to the disk.

How can I combine commands to do that while connected to the device over SSH?

  • I feel this lack of space everywhere may be a common situation... – beppe9000 Nov 19 '19 at 12:31
  • usually backup of /data is enough, but make sure to preserve selinux context too (besides uid/gid and permissions) – alecxs Nov 19 '19 at 21:04
1

We have authenticated login, encrypted archive, rsync and multiple transport protocols (including HTTP, WebDAV, FTP and SSH/SFTP) in one tool: rclone. It supports local filesystem and a number of cloud storage. Additionally it does have all of the four mentioned servers builtin. It means that all you need on both sides is rclone. No root access is necessary either. If you want to mount remote directory (using rclone mount), however, that requires FUSE support and root access on Android.

rclone does have builtin encryption i.e. files on remote will be created encrypted. Also there are multiple authentication methods supported including username/password and RSA keys (with SFTP), but those are usually unnecessary when encryption is already there.

A simple use case:

On server:

Run minimal SSH server if not already running:

~$ mkdir -p BackupDir/Encrypted
~$ rclone serve sftp --user <username> --pass <password> --addr :2222 BackupDir

On phone:

Create obscure password:

~$ rclone obscure <password>

Create configuration file (use rclone config for simple steps):

# rclone.conf

[PLAIN_TARGET]
type = sftp
host = <server_ip>
port = 2222
user = <username>
pass = <obscure_password>
#key_file = /path/to/.ssh/id_rsa

[CRYPT_TARGET]
type = crypt
remote = PLAIN_TARGET:/Encrypted
filename_encryption = standard
directory_name_encryption = true
password = <obscure_password>

Copy test file to server without and with encryption:

~$ echo Hi >test_file
~$ rclone --config rclone.conf sync test_file PLAIN_TARGET:/
~$ rclone --config rclone.conf sync test_file CRYPT_TARGET:/

SFTP supports modification timestamps, WebDAV and FTP both don't. However symlinks cannot be transferred as symlinks, those are either ignored or original file is copied or .rclonelink file is created. A way to copy symlinks and all metadata including timestamps, extended attributes (like SELinux labels), and ACLs is to throw tar stream at rclone rcat:

~$ tar --xattrs -cpvzf - test_file | rclone rcat CRYPT_TARGET:/test_file.tgz

Similarly you can get back the file using rclone cat. This will also largely reduce transfer time (even up to less than half) particularly if you have large number of small-sized files. Adding reasonable compression to tar may even reduce space usage and time, but high compression can go wrong too. However this way incremental sync won't work and you need to untar to use files on remote. It's also possible (by some scripting) to find only changed files first by running rclone check or by passing option --dry-run to rclone sync command and then add those files to tar archive.

I'm not sure if a zip of / is sufficient for backup purposes.

On Android rootfs / contains a number of directories and mount points - pseudo filesystems like sysfs, procfs and read-only filesystems like system and vendor partitions - which cannot be or don't need to be backed up. User settings, apps data and personal data is stored in /data partition. For details see Android Partitions and Filesystems and How disk space is used on Android device?.

So you can backup selected directories from /data or the whole partition at maximum. However, as mentioned by @alecxs in comment, if you intend to restore apps data (in /data/data), you'll have to minutely take care of files metadata - in particular UIDs/GIDs and SELinux contexts.

Say you want to backup whole /data partition, do:

~# tar --xattrs -cpvzf - /data | rclone -P rcat PLAIN_TARGET:/data.tgz

For ideas on how to dump complete partitions (block devices), see How to recover a deleted file from /data partition?

  • Thanks for the tip. Didn't know the program. I wanted to do a backup in such a way that I could restore both rom, apps and internal memory card. – beppe9000 Dec 10 '19 at 13:01
  • ROM is on /system, /vendor and boot partitions, apps and internal SD card are on /data. You can backup all of those using rclone without any constraint. – Irfan Latif Dec 10 '19 at 13:06
  • 1
    busybox tar does not support --xattrs tar binary from opengapps.org is required. only --selinux flag will handle RHT.security. so better use both flags (for higher compatibility) at least on gnu tar selinux flag is mandatory for proper restoring – alecxs Dec 15 '19 at 2:03
  • @alecxs right.. – Irfan Latif Dec 15 '19 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.