Google has a weird mix of app settings, device settings, and a website to manage location permissions. I've found that the easiest way to access most of it is through Settings → Personal content in the Google Maps app:
As you can see I've gone through those one-by-one to disable all use of my location info by Google. You might also need to go to https://myaccount.google.com/activitycontrols/location.
Android Device Manager still works though even though my device is not actively sending things up. To remove your device from ADM entirely, you have to remove it from your Google account via https://myaccount.google.com/device-activity. This will naturally also sign you out of Gmail and the like on your device.