So disconnecting WhatsApp from Google Drive didn't exactly work out because my chats were also in that Google Drive backup.

However, here is how I was successful in restoring only the chats. (No media).

1. Install WhatsApp and verify the number. 
2. Make sure your data connection is active. WiFi should be OFF.  
3. Click on restore backup (Entire backup, including Media).
4. It will restore the chats first. 
5. Then it prompts that the media will be downloaded over WiFi. Click yes (or next) and it takes you to your profile name and then you have all your chats. 
6. At this point, go to Google Drive and disconnect WhatsApp from the Setting - Manage apps. (Settings - Data and Privacy - Apps and Services - Apps with access to your account, Third-party apps with account access)
7. Connect WiFi, or force it to try to resume with data.
8. It would try to restore the media but it would fail.
9. It would ask you if you want to skip it. Might have to try to dismiss the reconnect prompt first. 
10. Click skip.

There you go. Only chats were restored. No media. It will prompt that media won't be restored anymore.
You can then reconnect aka relink the Google backup again in the WhatsApp settings.