It seems that the hardware erroneously regard the headset as attached at all times (to the 3.5" jack). This makes me unable to make calls without headset or speaker mode.
This is a hardware problem, but there is an app called toggleheadset2 that has a widget that allows you to override the hardware switch, toggling headset output off/on.
Plug in your headphones, power down the phone, power on the phone then remove the headphones. This worked on my GS3. on my GS3 the headphone icon was never on, but the phone was stuck in headset mode. Could not hear any calls, could not hear any music, UNLESS the headphones were plugged in. The speaker and Bluetooth icons were disabled so I could not switch to them.
I've been having this problem with my Samsung Galaxy Note III. It always thinks that headphones are plugged in. I even replaced the headphone jack and cleaned the contacts but the problem persisted. I found HF Button Widget (<=KitKat). As the name suggests it supposedly only works on KitKat or below, and I'm on lollipop. But it actually did work for me.
So with this I get a widget which I use to toggle the headphones. I practically always have it at the off-setting, which makes the phone behave as it normally would with no headphones plugged in, e.g. I can use the native speaker toggle in calls to switch between earpiece and speaker. I very rarely use headphones which means the problem for me is completely solved in that sense. When I do want to use headphones I have a one-click widget to use. One minor annoyance is that the headphones icon always stay in the status bar.