I have been researching about this for some time now, but I have to admit that I failed to get substantive information on the subject.
However, I have come to the conclusion, that this setting is hard-coded and difficult to manipulate (perhaps even with root) unless you're compiling a customised ROM from scratch with such modifications.
It may be a restriction by manufacturers perhaps the feasibility based on factors like Wi-Fi chip capability, CPU power, battery, etc. (from my observations, this limit ranges from 8-12 varying with devices)
The are some workarounds I propose, depending on what you're trying to achieve you could either;
- Set your Android device as a Wi-Fi repeater
- Or, share your device's internet connection to a desktop/laptop PC first, preferably via USB connection (i.e. USB tethering if possible) or other means then reverse-tether that connection to other clients from the desktop/laptop.
Using an Android device as a Wi-Fi repeater
- It is possible to use the Android device as a Wi-Fi repeater, through various methods most of which you can find here: Can we use an Android phone as a Wi-Fi repeater?
- The consequence is that you can make use of the second Access Point, to have other clients connect to the internet via a Wi-Fi AP broadcasted from the primary device.
NB: Watch out for battery drain and heavy resource consumption
Share Android device's internet connection to a desktop/laptop PC
- This concept is an indirect method which may demand other pre-requisites, i.e a desktop/laptop PC or other third-party software for hotspot share (may be freeware or shareware)
- Here, you need the Android device to have its internet connectivity shared to the primary device (i.e. PC) first, then the desktop (which is much more capable of accommodating a much larger number of clients than the Android device) can then share to other clients.
Some of the appropriate methods have been described here: Share WIFI from PC over USB and around the internet.
You can check this thread on XDA with a similar question: [MOD] Increase Max Users for Wi-Fi Tethering LG G2 LS980 [NON SPRINT ACTIVATED]