I tried to unlock my bootloader with fastboot but that didn't work
>fastboot oem relock XXXXXXXXXXXXXXXX FAILED (remote: 'root type is risk') Finished. Total time: 0.014s
So I thought that adb would work (if it's possible)
If your android device is modified in anyway such as a custom recovery, binary, or any other way. Trying to re-lock the bootloader should (should instead of would not all OEM follow protocol) usually cause the bootloader to issue a fatal exception like:
FAILED (remote: 'root type is risk'). This happens due to device custom modifications. It is a safety thing if you lock your bootloader with a modified device you will most likey Brick your device.
Technical snippets about verification from Android Developers:
AOSP Secure Features Verifying Boot Verified boot requires cryptographically verifying all executable code and data that is part of the Android version being booted before it is used. This includes the kernel (loaded from the boot partition), the device tree (loaded from the dtbo partition), system partition, vendor partition, and so on.
Small partitions, such as boot and dtbo, that are read only once are typically verified by loading the entire contents into memory and then calculating its hash. This calculated hash value is then compared to the expected hash value. If the value doesn't match, Android won't load. For more details,
Larger partitions that won't fit into memory (such as, file systems) may use a hash tree where verification is a continuous process happening as data is loaded into memory. In this case, the root hash of the hash tree is calculated during run time and is checked against the expected root hash value. Android includes the dm-verity driver to verify larger partitions. If at some point the calculated root hash doesn't match the expected root hash value, the data is not used and Android enters an error state.
The expected hashes are typically stored at either the end or beginning of each verified partition, in a dedicated partition, or both. Crucially, these hashes are signed (either directly or indirectly) by the root of trust. As an example, the AVB implementation supports both approaches.
Verification can fail either at boot time (such as, if the calculated hash on boot partition doesn't match the expected hash) or at run time (such as, if dm-verity encounters a verification error on the system partition). If verification fails at boot time, the device cannot boot and the end user needs to go through steps to recover the device.
If verification fails at run-time the flow is a bit more complicated. If the device uses dm-verity, it should be configured in restart mode. In restart mode, if a verification error is encountered, the device is immediately restarted with a specific flag set to indicate the reason. The boot loader should notice this flag and switch dm-verity over to use I/O Error (eio) mode and stay in this mode until a new update has been installed.
When booting in eio mode, the device shows an error screen informing the user that corruption has been detected and the device may not function correctly. The screen shows until the user dismisses it. In eio mode the dm-verity driver will not restart the device if a verification error is encountered, instead an EIO error is returned and the application needs to deal with the error.
The intent is that either the system updater will run (so a new OS without corruption errors can be installed) or the user can get as much of their data off the device as possible. Once the new OS has been installed, the boot loader notices the newly installed OS and switches back to restart mode."
Without knowing which device you have or what Android version your running. I will not be able to give a detailed conclusion tailored to your device but instead a simple generic one. Also be wary of anti-roll back if applicable.
That being the case obtain and download signed factory firmware. Use fastboot, download mode, or whatever your mode your device uses to erase and flash partitions. Depending on protocol erase and then flash the factory signed firmware thus returning completely to stock. Reboot the bootloader and re-issue the
fastboot oem relock XXXXXXXXXXXXXXX and you should be good to go.
To prevent unauthorized access to your personal data, locking the bootloader will also delete all personal data on your phone.
Keep's it simple no need to jump through a checklist of hoops to avoid flashing firmware to preserve User Data unless you cannot find signed factory firmware but that would be another question.