1

Where might there be a small program (possibly statically-linked binary) like fdupes or rdfind, to help me find duplicate files within the same partition/mount--checking only specified directories--and hardlink them?

My architecture: armv8l

root@localhost/storage/emulated/0/TitaniumBackup 2018-08-02,12:12:26# (16) uname -a
Linux localhost 3.10.108-ga044406f3ee #1 SMP PREEMPT Sun Jul 15 02:16:25 CEST 2018 armv8l

Assume I already have root shell access, with Busybox installed.

Filesystem               1K-blocks     Used Available Use% Mounted on
tmpfs                       944236      500    943736   1% /dev
tmpfs                       944236        0    944236   0% /mnt
/dev/block/mmcblk0p3         16048     2320     13728  15% /efs
/dev/block/mmcblk0p21       197472      428    197044   1% /cache
/dev/block/mmcblk0p23     11588512 11507720     80792 100% /data
/dev/block/mmcblk0p18         8080       92      7988   2% /persdata/absolute
/dev/block/mmcblk0p4          3952      548      3404  14% /cpefs
tmpfs                       944236      452    943784   1% /sbin
/dev/block/mmcblk0p20      3023760  1258796   1764964  42% /sbin/.core/mirror/system
/sbin/.core/block/loop08    157152   110660     46492  71% /sbin/.core/img
tmpfs                       944236        0    944236   0% /system
tmpfs                       944236        0    944236   0% /system/etc/permissions
tmpfs                       944236        0    944236   0% /system/priv-app
tmpfs                       944236        0    944236   0% /system/app
tmpfs                       944236        0    944236   0% /system/app/FaceLock/lib
tmpfs                       944236        0    944236   0% /system/lib
tmpfs                       944236        0    944236   0% /system/framework
tmpfs                       944236        0    944236   0% /system/vendor
/data/media               11588512 11507732     80780 100% /mnt/runtime/default/emulated
Improve this question
2
  • I doubt that there are many duplicate system files. What do you want to achieve - shrinking an Android system image or getting more space on your Android device?
    – Robert
    Commented Aug 2, 2018 at 11:55
  • Mainly concerned with stuff in /storage/emulated/0, away from the core system. Unrelated 3rd-party cloud/backup tools overlap too much (Lucky Patcher, SMS B&R, Titanium, etc) tripling+ old apk's & historical user data files not vital to operation. Except that left alone they eat too much damn space, causing wide Android failures routinely.
    – Marcos
    Commented Aug 2, 2018 at 12:36

2 Answers 2

Reset to default
0

SD Maid can help you find duplicates. You need to buy the pro version to unlock all features though (in the case of duplicates, deleting them from the app).

Improve this answer
6
  • Lots of apps can do that, eg. it4nextgen.com/duplicate-file-finder-apps-android , some of which I already had. Thanks, but the question is specifically about hardlinking (a sysadmin app or shell program that finds and knows where to run ln).
    – Marcos
    Commented Aug 3, 2018 at 9:00
  • Dammit I should read better. Don't know about an app, but if you want you can use fdupes and ln as commands in termux
    – SSS
    Commented Aug 3, 2018 at 11:53
  • Unless you know better, I've already looked around for an fdupes binary, and don't mean to try and deploy build-essentials or compile it myself, for armv8l. However, I might just write a shell script to scan file sizes and checksums, then link matches, schedule it after space hoggers finish, etc.
    – Marcos
    Commented Aug 3, 2018 at 15:22
  • termux has its own repository, you can do apt install fdupes from the app to install it from termux's servers
    – SSS
    Commented Aug 3, 2018 at 19:12
  • 1
    github.com/termux/termux-packages/tree/master/packages -- That's a list of all packages termux provides to download. fdupes is one of them
    – SSS
    Commented Aug 3, 2018 at 19:16
0

Until a better tool comes along, this script will find and hardlink duplicate files filtered out by extension, under given directories.

It's far from ideal, but others can customize it to their needs. As practically a one-liner I hate that it's incomprehensibly concise, but I had to deal with a few hurdles in a rather limiting BusyBox environment.

(cd /data/media/0/; (date; set -x; mount; df; fdupes -nAS1 -o time -r TitaniumBackup SmsContactsBackup; df) |tee fdupes.log |grep "\.apk" |sed 's/[][\\ ()]/?/g' |awk '{system( "ls -l "$0); system("touch -m "$2" -r "$1" && ln -vf "$1" "$2 ) }' 2>&1 |tee duphlinker.log)

  • The available fdupes for termux doesn't hardlink. So I only use its output identifying similar files.
  • Starting under /data/media/0/ (not /storage/emulated/0, since in modern Android that's a virtual mount not supporting filesystem links), I chose to recursively scan only directories TitaniumBackup and SmsContactsBackup, where I happen to know large duplicates form after their corresponding apps run.
  • I filter out all but .apk files.
  • Since some file paths contained escaped spaces--nasty when whitespace is needed as an unconfigurable delimiter later--with sed I replace all "\ " with the single-character wildcard "?". This allows a tiny risk of linking mismatched files, if they are improbably similarly named.
  • In awk I set the mod timestamps to that of the oldest file, then force form the hardlink, freeing disk space from the newer file.

The higher-level apps don't notice, and self-clean after uploading to my cloud eventually (if they have such features).

Now I'll just put that into crontab for Magisk, at about 4am daily. It runs in under a minute.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .