You shouldn't worry that much as it happened once. Just take care it doesn't happen often.
In short, LEDs have a long life span (20k+ hours?). So the concern isn't really the LEDs burning out (although after a long period of time, they will eventually start losing the amount of output and their brightness will dim).
The real concern is to not apply too much voltage to the LED and overheat it. Another concern would be overheating your battery (due to heat dissipation from draining the battery from powering the LED).
In short, if you're only using the LED for time periods less than half an hour, you should be okay.