The formats
tar
This is essentially snapshots of all partitions each in single file with extension tar. If you make two backups of the same rom with little difference in user apps, two snapshots of tar files are created, consuming more data on your sdcard.
dup
(dedupe
)
This backup format backups up individual files in the partitions renamed uniquely depending on some hash algorithm. The major advantage of this is it takes up less space on sdcard. If you make two backups of the same rom with little difference in user apps, the second backup only contains changed files. And when you delete the first backup from recovery, the common files are never deleted, so you don't loose your second backup. But it creates many files on sdcard, thereby many indexes.
tar + gzip
It is same as the tar format with gzip compression applied to save some space on sdcard.
Which one should you choose?
Which one to use is upto your choice. I wouldn't recommend dup format as it takes considerable time during operations. But if you donot backup/restore frequently and concerned about the space on sdcard, u may use dup.