No downside to it (barring some scenarios)
HDR increases the dynamic range between the darkest and brightest colours in your photo and there's no downside to it apart from lag (that may be fixed by using camera apps that respond faster -you would need to experiment)
But it's not a silver bullet to increase the quality. Some situations where it spoils rather than help :
If the picture you wish to take isn't already brightly lit with a bright spots and dark ones (like shadows) , it's better to avoid HDR. In this case, HDR photos look artificial. As Izzy pointed out in comments, pictures that don't have much contrast between dark and bright (low dynamic range) don't come out well. HDR can only enhance range not create . Try taking a picture indoor with subdued lighting with HDR to see how poorly it comes out.
If you or the object is moving, the lag may cause blurry photos (HDR takes 3 photos – normal exposure, under-exposure, over-exposure – and combines them. (In professional cameras it takes even 7 photos). If the object moves in this process , it becomes evident as blur. This is called ghosting (anti-ghosting algorithms may be able to compensate, but they have their limits). Same result if you move while taking the shot.
Also if your picture has bright colours already like close up of a bunch of flowers in a garden, it adds artificiality.
For a technical understanding see this from our sister site What is HDR
You may feel that HDR is not kicking in when it is required, but then it is the software which decides when it should, so better leave the judgement to it ! (Setting on Auto)