Let me start with an example, Delta's airline app called "FlyDelta" has an option in it settings to enable/disable "Get check-in reminders & status notifications". My question is, if I enable this "Get check-in reminders & status notifications" will the app from that point on run in the background at all times (even after restarting the phone or killing the app)? I'm a bit confused on how notifications work. I know that Google has a very nice infrastructure to push notifications to an app, but does the app has to be running in order to consume the data? any help explaining this specific app behavior will be greatly appreciated.
Polling for updates
It depends on how the app is written. Most commonly, the app uses Android's
AlarmManager to run itself at intervals (say, every hour) to check for updates. Some apps might do this to keep themselves in sync with server data regardless of whether notifications are turned on or off in the app, and may (or may not) have another setting to disable syncing completely. Other apps might only run this periodic check if notifications are turned on, and only sync when you start the app yourself.
However it controls that, you should expect that when the
AlarmManager starts the app in the background, it runs for as it long as it takes to get any new data from the server, and then stops running when it's finished. It will then not run until the
AlarmManager starts it again in another hour (or however long it's set to wait). There's (usually) no need for the app to be running in the background the whole time, though of course a badly written app may do this.
Cached background processes
Don't forget that Android keeps background processes in memory even when they've stopped running, unless/until another process needs to use that memory. If the app is 'stopped', it's not using any resources, not even if Android is keeping it in memory.
Google Cloud Messaging
Apps can use Google Cloud Messaging (GCM) to be 'pushed' updates from servers (i.e. from "the cloud"). Instead of being started at regular intervals to check for data, they're only started when new data are available. This means they use less power, because they're not being started unnecessarily; and less data, because they only talk to the server when they know it has new data.
As above, an app might use GCM to keep its data in sync with the server, even if notifications are turned off. It's an implementation detail of the app: it shouldn't affect its observable behaviour, just how much power and data it uses.
Android itself keeps an open connection to Google's servers. There's only one such connection on the whole phone, so installing a new app that uses GCM doesn't use any extra resources. When the Android system receives a GCM message for a particular app, it starts that app in the background to give it the message: the app doesn't need to have been running already.
The app then runs, and reads the message. The size of GCM messages is severely limited, so the app may still need to contact the server to find out more. For example, the GCM message might just say how many new emails there are, and the client then needs to check the server to download the new emails. For flight details, it may be that the message only has a gate number or an updated departure time, so the app might not need to contact the server for more details. As before, when the app has finished processing the update, and it has created a notification if necessary, it can stop itself: it doesn't need to keep running.