Missing space on internal storage

Question

I have a GT-I9195 running Lollipop 5.1.1 (currently PAC-ROM) which in theory has 8GB of internal memory. However, I can't seem to track down what exactly has happened to all that space. With a paltry 100 or so apps (including system essentials) installed, the phone starts reporting it's out of space until I move apps to the external MicroSD card. The problem persisted even through the custom ROM installation and subsequent recent ROM update as well as a few factory resets/cache wipes/etc. It's been an issue for nearly 8 months now, and periodically I've trawled the Internet for suggestions and clues but without making any real progress.

It seems to me as if a large chunk of space on the phone's memory is simply inaccessible and unused. Is this possible? How can I delve deeper into what's going on?

I've included the output of a df command in terminal emulator below. I notice that there seem to be duplicates of the internal memory /storage/emulated/0 and /storage/emulated/legacy (though I think one is just an alias of the other), and that they are both the same reported size as /data.

Storage Analyser reports 541.93MB of that 5.3GB is used by accessible files and 3.59GB used by "android/inaccessible files". This adds up to the 4.1GB which df claims is in use, but I can't understand what that 3.59GB is when every app which can be put on the external SD card is already on it, and I thought that the /system partition contained the OS itself and was not included in /data?

I've tried to get ES File Explorer to do its SD card analyst trick on the /data folder but without success, so it's not easy to probe it and find out how big each subfolder is.

I'd really like to get to the bottom of this... can anyone offer a suggestion as to what to try next?

Edit: also of interest to me is that I do have more than a gigabyte of space on the internal storage (/storage/emulated/0), but this is apparently unavailable for installing apps on. It holds DCIM and other random storagey things, and naturally freeing up space here doesn't allow me to install any more apps on the phone memory.

Filesystem               Size     Used     Free   Blksize
/dev                   663.7M    48.0K   663.6M   4096
/sys/fs/cgroup         663.7M     0.0K   663.7M   4096
/mnt/asec              663.7M     0.0K   663.7M   4096
/mnt/asec/org.mrpdaemon.android.encdroid-2     6.0M     4.8M     1.2M   4096
/mnt/asec/flipboard.app-2     9.0M     7.4M     1.6M   4096
/mnt/asec/com.netflix.mediaclient-2    33.1M    31.9M     1.2M   4096
/mnt/asec/com.tecarta.bible-2    17.0M    15.3M     1.7M   4096
/mnt/asec/com.pisanu.scrabblechecker-2     4.0M     2.7M     1.2M   4096
/mnt/asec/com.shazam.android-1    27.1M    25.5M     1.6M   4096
/mnt/asec/com.hutchison3g.planet3-2    12.0M    10.3M     1.7M   4096
/mnt/asec/com.trello-2    16.0M    14.2M     1.8M   4096
/mnt/asec/com.whatsapp-2    22.0M    20.4M     1.7M   4096
/mnt/asec/com.audioaddict.di-2    33.1M    31.1M     2.0M   4096
/mnt/asec/com.evernote-2    28.1M    26.4M     1.7M   4096
/mnt/asec/com.mxtech.videoplayer.pro-2    25.1M    23.8M     1.2M   4096
/mnt/asec/com.mxtech.ffmpeg.v7_neon-2    19.0M    17.7M     1.3M   4096
/mnt/asec/com.ideashower.readitlater.pro-2    13.0M    11.1M     1.9M   4096
/mnt/asec/com.skype.raider-2    80.3M    78.9M     1.4M   4096
/mnt/asec/com.spotify.music-2    39.1M    37.5M     1.6M   4096
/mnt/asec/org.wordpress.android-2    11.0M     9.9M     1.1M   4096
/mnt/asec/com.Slack-2    31.1M    29.3M     1.8M   4096
/mnt/asec/jackpal.androidterm-2     2.0M   604.0K     1.4M   4096
/mnt/asec/com.mictale.gpsessentials-2    15.0M    13.6M     1.4M   4096
/mnt/asec/com.adobe.reader-2    17.0M    15.7M     1.4M   4096
/mnt/obb               663.7M     0.0K   663.7M   4096
/system                  1.4G   759.7M   714.5M   4096
/data                    5.3G     4.1G     1.1G   4096
/cache                 196.7M     4.5M   192.2M   4096
/efs                    13.4M     4.2M     9.2M   4096
/persist                 7.9M     4.0M     3.8M   4096
/firmware               64.0M    49.7M    14.2M   16384
/storage/emulated      663.7M     0.0K   663.7M   4096
/storage/sdcard1        59.5G    57.2G     2.2G   131072
/storage/emulated/0      5.3G     4.1G     1.1G   4096
/storage/emulated/legacy     5.3G     4.1G     1.1G   4096

Edit2: I finally did some digging in /data by manually selecting directories and querying them with ES File Explorer. Thankfully this didn't take too long before I found the main culprits:

data/media           1.45 GB
data/data            1.11 GB
data/dalvik-cache    1.17 GB
data/app             642.57 MB
data/user            1.11 GB (this appears to be a symlink of data/data)

Investigating data/media, I discovered a 1GB backup file created by ClockWordMod recovery, presumably when I first flashed the ROM. Moving it to my SD card has apparently recovered that space in /data, which is fab. This has, presumably, been the main source of my problem - that is, assuming this space is actually available for installation of apps now (not easy to check with this ROM other than by installing apps until I run out of space).

As for the other big directories: I can believe that data/app is the total size of my installed apps, and I guess there's no getting around the dalvik cache being so huge. I'd really love to dig into data/data and see what exactly is using so much space in there, but in ES File Explorer I have to manually select each folder and click properties to find out how big a directory is - a task for another time as I've been unable to find a utility to do this for me.

Answer 1

So this is finally sorted. Here are the tools I used:

Initially, in Android Terminal Emulator, I used the df command (directing the output to a text file to post here: df > /storage/emulated/0/df.txt) to get an overview of the size and usage of each partition in the filesystem.

I later discovered FreeSpace which told me the same information but with a GUI and was somewhat easier to use. Doing this pointed to /data as the culprit.

I then used ES File Explorer to analyse individual directories. However, a more efficient way would have been to use the excellent DiskUsage (Google Play Store, F-Droid) app (thanks Izzy) - I've since pointed it at /data and it shows the exact size of every subdirectory in the entire tree (this does require root).

Doing this I discovered there was a large (1GB) backup file apparently created by ClockWorkMod Recovery when I installed the ROM (though with an odd timestamp, but that's another story). Using ES File Explorer again, with root access, I was able to move this backup file to my external SD card, freeing up space for downloading and installing apps.

Since doing this I've downloaded every large game app I can find and installed it to try to verify that the space is really available to install apps, and it does seem to have worked.