Your earpiece speaker may be dead. When you put your phone in speaker mode it uses different speaker(s). Some people use a bluetooth earpiece to get around this, but if that's not something you want to do then you may want to check and see if your phone is still in warranty.
Also, depending on your phone you can try taking a small brush and cleaning around your earpiece speaker. In some cases dust and other debris can collect and muffle the speaker, and considering how tiny these speakers and their speaker grills are (again, depends on your phone) it doesn'take much dust or debris to make it hard to hear. When i worked for Verizon this was a common problem that we encountered on a daily basis. Hopefully, that's all it is.