As I've found in this bug report, you can watch the current open files by running this command in a second shell:

while true; do lsof | grep /data; sleep 0.2; done

This checks every 0.2s which files are still in use on /data. In my case it was chainfire's SuperSU, which had a logfile opened.

As I've found in this bug report, you can watch the current open files by running this command in a second shell:

while true; do lsof | grep /data; sleep 0.2; done

This checks every 0.2s which files are still in use on /data. In my case it was chainfire's SuperSU, which had a logfile opened.

Additionally, I've found out that a 2nd SDCard partition was mounted to /data/sdext2:

mount | grep "/data"

This was due to an app "Link2SD". With this command I was able to figure out, where the mounting happens:

adb shell
# make system partition writable
mount -o remount,rw /system
grep -R "/data/sdext2" /system

I've commented the "mount" commands out and restartet my phone. Now encryption works fine.