Don't save your credit card info. (Edit: Just tried, you can't enter info without it being saved. Ugh.) You can set up a PIN in the Market but that's easily worked around by clearing the data. Google Wallet (Checkout) doesn't appear to have any security settings beyond needing to log in, though obviously saved credentials work on Android.
I think then that your only option (besides using another market) is to use full encryption on the device such as is present in Honeycomb in combination with a lock screen and not being rooted. Which doesn't rule out exploits, but if there was no risk of your CC being used nefariously then it wouldn't be usable by you either :P. Of course, if your phone is stolen you could just cancel your cards and/or let your credit card company know about the risk or actuality of unauthorized purchases.
Google will also help you fix it:
Account Hijacking
If you previously created a Google Wallet Account, and an unauthorized person has accessed your account, please contact us immediately. Our fraud protection policy covers you against any unauthorized purchases that were made with your Google Account. If we find that an unauthorized purchase was made, we'll make sure you aren't charged for it.