In the regular XAuth/RSA authentication scheme both client and server are mutually authenticated using RSA certificates during Phase 1 of the Internet Key Exchange protocol (IKEv1) that is used to negotiate IPsec connections. In a second authentication step the client authenticates itself using XAuth (e.g. with username/password, but there can be other credentials), which is desribed in draft-ieft-ipsec-isakmp-xauth and an extension on top of IKEv1/ISAKMP (often called Phase 1.5 as it comes, together with Mode Config, between Main/Aggressive Mode and Phase 2, Quick Mode).
With the hybrid XAuth/RSA scheme, which is described in draft-ietf-ipsec-isakmp-hybrid-auth, the client does no authentication during Phase 1 of the negotiation. Only the server does so during that phase, allowing the client to verify that it is talking to the right VPN server when it continues to authenticate itself using XAuth. This simplifies deployment as no client certificates/keys are required. And compared to the commonly used XAuth/PSK scheme it prevents man-in-the-middle attacks, which are possible there due to the publicly known pre-shared secret (at least in larger deployments).