If you've read around, then you'll already have read that the idea of "free RAM" is a bit of a nonsense for Android. PC operating systems take an app out of RAM as soon as you quit it, which leaves empty space that the PC probably has to spend power on filling again afterwards - probably with the same app next time you run it. Android tries to avoid this waste by leaving apps in memory after you stop using them or after they finish running, so that they're ready to run again next time. If it needs more RAM for an app, only then does it get rid of these cached processes.
Android will first get rid of cached processes, based on how soon it thinks it's going to need them again. If it still needs more RAM for the foreground activity, it'll get rid of background services (which sync your email etc. but aren't directly visible). If it still needs more RAM, it'll get rid of activities that are directly accessible but aren't currently visible, such as the launcher, the activity you'll go back to when you press back (maybe even in the same app as the activity that's running), or the greyed-out activity that's partly visible behind a dialog. This way, even if you have no free RAM, Android can free up quite a lot of RAM for the foreground activity: all the RAM on the device, except that needed by Android itself.
I suggest to work out the best setting using trial and error. Without more knowledge of the app, I can't predict the effect of having the setting too low: perhaps the AI will make worse moves, or take longer to decide. If the setting is too high, you'll notice that when you leave the chess game you have to wait for the previous app to launch: this is particularly noticeable when it's the launcher, and you get a spinning progress indicator instead of the list of apps.
If this were one of my apps, I'd consider this a bug. It's bad form to give the user a setting with no way to know what value to set. The app has more information about how much RAM the device has and can make available than you do: it should decide itself.