It's not possible using the power button long press for the Device Options menu in 4.0.4 on SG2. It always reverts back to "Vibrate" when changing Silent mode, albeit it is possible to press the volume down until it goes into vibrate, and then press volume down again. This takes a bit of time, though, and doesn't work unless you're past the screen lock. IMO, in a perfect world, this should only take one action.
Conclusion
Wait for a version update above 4.0.4 and hope they fix this BS. (Also, I friggin' hate people who think their phone is silent during meetings or seminars, and their phone just keeps buzzing and buzzing and buzzing. It's #*@!ing Android trolling!)
Hack
The only hack I've found to make it work with the power button long press is going to Setting > Sound > Vibration intensity and setting ringtone vibration to zero. However, I'm pretty sure that turns off vibration globally, and you'll still see that annoying icon for "silence with vibration" when you use Silent mode trough the power button. :(
Fastest way to mute
Wipe off the screen lock. Flip down the curtain of the notificaions menu, then double tap the sound icon. That will bring you from sound to vibrate, and finally to mute. Or get a widget for muting.