Up to Android 2.x, in the labels view the Gmail app of my Nexus S displayed the number of unread messages next to labels with unread mail. I recently upgraded to Ice Cream Sandwich 4.0.4 on the same device and now Gmail no longer displays unread counts for labels unless I access the list of messages for a certain label. Is this a new Gmail feature? Is there some way of viewing those unread counts in a way similar to Android 2.x?
Yes, at least 4.0.4 stock image on my Galaxy Nexus does, though if more than 999, it just displays 999+ (same as inbox).
My Galaxy Nexus with stock ICS 4.0.2 does show unread counts on my labels, like this when I click the labels icon at the bottom of the screen:
Possibly more important than the version of Android you're running, is the version of GMail that you're runnning. The GMail app was last updated on 4th April 2012, I'm running v4.0.5 which I'm pretty sure is the latest version. Check which version you have by going into Settings -> Applications and finding GMail in the list. Visit the GMail page in the Play Store (on your phone) to update, if not.