It could be that Telegram may misbehave when find those files missing. Anyhow. You can use command-line for the job. Setup adb in PC and execute this command
adb shell am force-stop org.telegram.messenger # this would force-stop Telegram app.
adb shell rm -rf /sdcard/Android/data/org.telegram.messenger/* # * means everything - all files and folders. Replace it with the directory name and enclose it in "" if the files to be deleted are inside a sub-directory.
Alternatively, you can do it using a terminal emulator app:
- Force-stop Telegram app from Setting → Apps → Telegram → Force stop.
adb shell from the said second command and run it on terminal app.
Another alternative is to use a third-party file manager and see whether it hangs. I would suggest using even an automation app. You can use my answer here for further info. (Tip: Use only the actions not the profiles there. Actions can be performed without a trigger/profile.)