Navigate to Settings/Apps/Advanced/Default Apps/Digital Assistant, then change from Voice to Google speech recognition. That should replace the mic symbol with the phone symbol.
You could try denying the Google app's mic permission or both.
I had the same problem with Google Voice Access. I came across these instructions, which seem to be for Google Assistant. Executing them did not seem to affect the desired change in Voice Access, but I think they might work for you with Assistant.
The configuration is performed in the main Google app. After opening it, I clicked "⋯ More", then &...
It was possible in earlier versions of Android. And I find the same problem, too. Using the third-party apps seems to fix this problem.
Privileged apps have higher priority than ordinary apps.
Apps with visible foreground UIs have higher priority than background apps.
Apps capturing audio from a privacy-sensitive source have ...