You can easily update your Galaxy S4 to Android 5.0.1 Lollipop without root. As you have shared the screenshots, it shows your device is up-to-date but your region might have not received the latest update yet.
But you can check the sammobile firmware page over here.
It clearly shows that Lollipop 5.0.1 has been released for those regions, Now If your region is listed over there or not. You can simply download the firmware and flash it using Odin3. (If you don't know how to update using Odin, I've written a step-by-step guide on my blog).
I hope that the above instructions helps you to update to Android Lollipop. Now You also have said about Android Marshmallow, Now officially it is not available for your phone model and I am pretty sure it won't be in the near future as Samsung does not support their models too long and Galaxy S4 is now quite old.
But doesn't mean you can't have Android Marshmallow on your phone. In order to have it, you need to root your phone and install Custom ROM such CyanogenMod 13.
Let me know if you want to know more in details and I hope you've found my answer useful.