I'm started my LG Optimus One in System Recovery Mode. Vol Up/Vol Down buttons works - moving up/down through the menu. But i can't start menu item, for example 'reboot system now', or 'apply sdcard:update.zip'.
What type of System Recovery do you have? On my P500 everything is fine.
In AmonRA Recovery there are such bindings:
VolUp/VolDown - moving through menus
Back - previous menu level, cancel action
Menu - select menu item, approve action
Also I've noted that from some time back button doesn't work every time as I press it. I think it is hardware problem. Maybe I need to disassemble phone and make it maintenance.
Normally you click the Power button to select items. That's how my Galaxy S works, anyways.
Edit: According to this, try using the Send Call button. And End Call for cancel. Or according to this, use the Camera button. I guess it depends on what recovery you have installed. Try selecting something that won't do any harm -- a further menu, for example. Not "factory reset" :P
if you're use the same custom recovery, according to this, on P500 to select item you press Menu button.