How to extract ab files
There is an open source project under the Apache 2.0 license, written by Nikolay Elenkov that will allow you to extract the .ab in to a tar file.
java -jar abe.jar unpack <backup.ab> <backup.tar> <password>
Just for reference of others, here is some background on the .ab file format.
The Android Backup (*.ab) file is a compressed TAR file. It is compressed using the DEFLATE algorithm. On top of that, there can be AES encryption used. This is determined when you create the backup, if you enter a password then the backup is encrypted, otherwise; there is no encryption, it is only compressed.
The HEADER of the file is a little different than a normal DEFLATE archive. It contains information about the backup and looks like the following:
The first line is the "Magic" line. The next line is the version of the Android Backup file format. The next line is a boolean (true or false, 1 or 0) indicating if the file is compressed. The last line is the type of encryption. This example is not using any encryption. If there was a password, the line would read "AES-256". After that is the encryption cipher. If no password, then the DEFLATE "archive" starts.
It is compressed using the Java Deflater. Which, from a developers perspective, causes issues if you want to use anything besides Java to extract it. I haven't been able to find anything that can deflate it using the same algorithm, even though all that I have found (for like C#) are supposed to follow the "SPEC".
If you are not sure how to really use that (which is beyond the scope of this answer) Droid Explorer since v0.8.8.7 (available here) allows you to do exactly this, and more, right from Explorer. You can read more about the features on my blog (yes, i know, shameless plug. I do that when it fits the question)