Technically, yes, you can build the "plain version" of Android: http://source.android.com/source/initializing.html
However, it is difficult to do this for all but a handful of devices (official "Google" developer devices, such as the Nexus line of phones) because every Android phone requires its own drivers and these are not freely available.
This is where projects like CyanogenMod step in: they take the vanilla build of Android, add a few minor features to it, and then try to release it for as many devices as possible. Typically way before any manufacturer gets around to updating the phones (if ever). CyanogenMod 9 is the ICS version of Android, and is tba.
In short, the difference between HTC and Google is that HTC takes the vanilla Android version and then "skins" it with HTC Sense. This is one of the many reasons there is a bit of a delay between when Google release a new version of Android (for example, ICS) and when the manufacturers get around to doing it, because they have to fix their changes/skin to the new version of Android.