You can install Cyanogenmod or any number of custom roms, but it sounds like you'll need to develop a specific rom for your situation.
If you only want the user to run a single app, I would imagine you'd have to develop an app that declares itself as a Home Screen/Launcher, and have your ROM only include that app as the main Launcher.
This is basically how Barnes & Noble made the Nook Simple Touch. Their own Launcher is very limited and doesn't let you install apps or see anything that looks like Android. As soon as you get root via an exploit, you can easily install your own launcher that gives you access to the standard android interface and turns the Nook Simple Touch into a fully fledged (if limited) tablet.
A specific implementation is outside the focus of this stackexchange site, but you can get programming advice at stackoverflow.com