I'm using a flashlight app on my Xperia L phone. I accidentally turned on the flashlight, but noticed it only 4 hours later. I turned it off then.
Is this keeping the flashlight on for a long period of time harmful for my phone or flashlight?
You shouldn't worry that much as it happened once. Just take care it doesn't happen often.
In short, LEDs have a long life span (20k+ hours?). So the concern isn't really the LEDs burning out (although after a long period of time, they will eventually start losing the amount of output and their brightness will dim).
The real concern is to not apply too much voltage to the LED and overheat it. Another concern would be overheating your battery (due to heat dissipation from draining the battery from powering the LED).
In short, if you're only using the LED for time periods less than half an hour, you should be okay.