Activities don't run in the background
In Android, activities (that is, the part of the app you can see) never run in the background. They can only run (and use battery power) while they're on the screen. The activity stops running regardless of whether you use home or back to leave it. The only difference is what data Android asks the app to save, so neither option is "the right way". It just depends on what you want to do.
If you use home, Android leaves the app in the same state, so that if you come back to it later (e.g. through the recent apps list), it'll still be in the same state you left it: on the same screen, with the same stuff shown. For example, if it's an email app, and you were looking at one email, then it'll remember which email that was, and show you the same one.
Eventually (after about half an hour), Android concludes that you're not coming back to the app, so it resets this state: next time you start the app, it'll start from the front/main screen. To continue the example, the email app will forget which email and folder you were looking at, and show you the inbox.
If you use back, you're telling Android that you don't want to come back to this view. It'll destroy the information about what you were looking at right away. Next time you start the app, it'll show the front screen (e.g. the inbox).
As others have said, apps can control the behaviour of the back button: for example, web browsers use it to go back in the browser history. What I've described is the default behaviour of the back button, and developers are urged to keep the behaviour like that to avoid being confusing.
Cached background processes
Whichever method you use, Android will leave the app in memory (but not running) for as long as it can. This is to be more efficient. When you come back to the app, if it's still in memory, Android can run it again right away; if it isn't still in memory, then Android has to spend time and energy loading the app from storage again.
In old Android versions, apps left in memory in the background this way were included in the list of "running apps". This is a little confusing for users - it makes people think the app is really still running - so newer versions call these apps "cached background processes", to make it clear they're only cached, not running.
What about background apps?
Earlier on, I said that activities don't run in the background. So how does your email client check for mail? As well as activities, apps can have services. Services don't have any GUI for you to see or interact with, but they do run in the background. Usually, a service will only run infrequently, such as to check mail once an hour, but it's possible for the app developer to run the service all the time, draining your battery.
Leaving an activity with back or home doesn't change how Android treats any services from the same app: the service can continue to run, or be triggered later at a given time (next time the mail check is due).
In summary, it doesn't really matter whether you use back or home: it only changes what the app shows you next time you run it. It doesn't have an effect on battery use. Neither of them corresponds to "exiting" a program on your PC.