Android keeps closed applications on memory as long as possible so that when you return to the application it will still be on memory and will load faster than loading from the internal memory or SD card; but if another applications needs memory, unused applications and background applications will be automatically force killed to give way to the foreground activity.
In short, Android is not Windows, don't treat them the same.
You should only force close misbehaving applications, not for freeing memory; without a dedicated task killer, you can do this from Settings > Applications > Manage Applications > *name of application* > Force Stop