I recently acquired my Nexus 5. While I'm absolutely in love with the device, I haven't been able to find a way to set a custom ringtone for specific contact groups.
As a Nexus 5 owner I was not able to do without a specialized 3rd party app (I used Group Ringtone: http://www.appbrain.com/app/group-ringtones/com.appventive.groupring). This feature seems to be missing in Android.
Try downloading a app from the play store like ringtone maker, it worked for me.