Work-around for process substitution in mksh

Question

An extremely useful feature of Bash, known as process substitution, is missing in the Android shell, mksh. This is highly unfortunate as it prevent you from doing things like:

diff <(sort list1) <(sort list2)

The mksh site has marked this as a "future plan" here. So my questions is:

Are there any work-arounds for this? (And what are they?)

Answer 1

Apparently the only (?) way to do this, is by using a named pipe like this:

mkfifo myp1 || exit
mkfifo myp2 || exit
sort list1 >myp1 &
sort list2 >myp2 &
diff myp1 myp2 
rm -f myp1 myp2

This need to be put into a mksh shell function in order to be of any real command line use. Another tricky part seem to be that AOS has implemented some kind of timeout that kills or messes up the pipe, if not used within a few seconds. (Reason unknown.)

Answer 2

We just figured out how to do this for the Desktop Unix case. On Android, you’ll need a directory to place temporary FIFOs at (any will do, such as /sqlite_stmt_journal in Android 2.x and /data/data (if you have the rights to write there) in newer ones). You’ll also need mktemp and mkfifo. (cat is an mksh builtin these days, but on old Android you’ll need to add that or a newer mksh version; they all work down to at least Android 1.5)

function die {
        print -ru2 -- "E: $*"
        exit 1
}

function psubin {
        local stdin=$(cat; echo .) pipe

        pipe=$(mktemp /tmp/psub.XXXXXXXXXX) || die mktemp

        # this is racy
        rm -f "$pipe"
        mkfifo "$pipe" || die mkfifo

        (
                # don’t block parent
                exec <&- >&- 2>&-
                # write content to FIFO
                print -nr -- "${stdin%.}" >"$pipe"
                # signal EOF to reader, ensure it’s not merged
                sleep 0.1
                :>"$pipe"
                # clean up
                (sleep 1; rm -f "$pipe") &
        ) &
        print -nr -- "$pipe"
}

diff -u $(sort list1 | psubin) $(sort list2 | psubin)

Answer 3

You need a shell such as ksh, zsh, or bash for process substitution. I assume the diff command @user1147688 is refering to is from busybox. Process substitution doesn't work with busybox apps. Busybox diff handles named pipes differently than diffutils diff. After a little more testing, I found I was only able to use process substution with busybox diff after creating the /tmp directory with this supersu command:

su -mm -d -c 'mount -t tmpfs -o rw,uid=0,gid=0,mode=1777 /tmp /tmp'

This creates a temporary for all users to use. For some reason, busybox doesn't use the TMPDIR variable. Alternatively with zsh, you can can use busybox diff like this:

busybox diff =(sort ./a) =(sort ./b)

This is like diff <(...) <(...) but it uses temporary files instead of named pipes. Zsh will use whatever read-writable temporary directory you assign to TMPDIR. Using TMPDIR=/sdcard won't work here though since you can't change ownership or permissions on /sdcard files.

Diffutils diff works without any issues using any kind of process substitution.

Here's a function that implements something like diff with process substitution. You can use this with busybox diff and any modern sh-compatible shell that supports arrays, such as ksh, bash, zsh, or mksh. It will work assuming HOME is set to a read-writable location, such as /sdcard on Android.

# usage: diff2 COMMANDS1 -- COMMANDS2
diff2() {
        local i=1 j k cmd1 cmd2 list1 list2
        cmd1=()
        cmd2=()
        while (( i < $# )); do
                eval j="\$$i"
                if [[ $j = -- ]]; then
                        k=$i
                        break
                else
                        cmd1+=("$j")
                fi
                let i++
        done
        shift $k
        cmd2=("$@")
        list1=$HOME/diff$RANDOM
        list2=$HOME/diff$RANDOM
        eval "${cmd1[@]}" > $list1 2>&1
        eval "${cmd2[@]}" > $list2 2>&1
        diff $list1 $list2
        rm $list1 $list2
}

You can add pipes, |, to your COMMAND1 or COMMAND2 as long as you quote them or backslash them.

The way this works is by splitting up the input into two arrays with -- as the separator for the commands. Some eval abuse helps separate the input and is needed to evaluate commands that use pipes.

The function could be further modified to include options for diff, either utilizing a third array or parsing with getopts / GNU getopt. A third array with another -- separator would probably work best to avoid having to use GNU getopt for long options.