Precisely. In fact, there is a plethora of evidence suggesting true caller is a privacy invader.
Just as Izzy mentioned, it's clear that the phonebook databases are uploaded to their servers even without consent from the users whose data end up leaking into their servers.
Truecaller uses "crowd-sourcing" to get data to their servers. In this article by Shaik Sharoz, the author explains the logic behind crowd sourcing:
The thing happens in this Crowd sourcing is they collect the data from you and shows to you in return. Suppose When a user installs the true caller app in his\her mobile, After installing it all the contacts which are present in the users mobile will sync with the servers of True caller app.
Consequently,
This brings out a privacy issue that your name is appearing in the search results to others very openly without permission.