No, you cannot change the resolution of Android in VNC in a standard way. However, some Android phones have additional functions to support HDMI (which results in running Android at a higher resolution). If you can somehow tap into that functionality in conjunction with VNC, then you might be able to get a 480p, 720p, and 1080p option. But this is something that would be highly customized to a single phone as all of the different HDMI implementations seem to vary tremendously. One other option that would be yet another one-off possibility (with some custom work) would be if your phone supports WebTop functionality. If so, perhaps you could tap into this.
However, there is no third-party app that does what you're wanting. To do this, you'll have to develop or find somebody to develop this highly-customized solution.
As other have suggested, you're probably not going to get much better than changing the LCD Density, which clearly isn't what you're after.