What I do is simply create a separate folder in Internal Storage, which contains all my 'pics for Contacts' -- SD content always tends to change OR you change the SD card throughout years/lengthy periods of time. Whenever I get a new phone, I simply transfer/copy the contents of my 'pics for Contacts' folder, which is located in my Internal Storage folder, via Bluetooth to the new phone.
A SIMPLIER WAY of course would be to just start saving your Contacts in your Google account under your main account. This way, any contact, together with its contact photo, will be saved in Google Cloud. You stated that this does not seem to work for you. Everytimre I buy a new phone and connect it to my main Google account during initial phone setup, my Contacts and their pics transfer onto my new phone without a problem.