Being "logged in" is a misconception. Even when you are "logged in" when you use a browser, this is handled by the storage of a cookie with an authentication token stored in it. On the device is sort of the same, except it is not a "cookie". When you set up your account, an authentication token is requested and stored on the device. A new token can also be exchanged at other times, but you are unaware that it even happens.
When the applications, like gmail, go and check for new mail, they use that token to tell the gmail servers that you are "you". The reason you can't "sign out" is because then you would not be able to check for new mail, get application updates and other things like that that happen in the background. If you were to sign out, then every couple minutes you would have to put in your credentials so your device could check for updates and new mail.
A large set of the services built-in to android use your authentication token. Even when you create, edit, or delete a contact on your device, because it syncs that data to your account on google servers. Calendar appointments, gTalk, Google Voice, the search widget, voice-to-text, push notifications (for just any service that uses C2DM), and any other service that may show up under your google account in Accounts & Sync
can and will need this authentication token at any given time.
It is more like you are logged in to your PC (Windows, Linux, OSX) then logged in to any particular google service.