For it to work, after creating the. nomedia file, you must move the media file out, then back into the folder.
I used ES file explorer to create the .nomedia file. Once you create one, just use copy/paste to spread it around.
Also in order to avoid the stress in future, it's better to place the .nomedia file in newly created folders so that all future media file that end up in it won't show up in Gallery.
I found the nomedia file in Evernote folder which means they too foresaw this problem :-)
Edit: I came across another solution.
After creating the ".nomedia" file, instead of manually moving the media files to and fro, you could remove and reinsert the SD card in order to force the gallery to rescan folders.