There are a lot of alarm clocks around, and B"H one of them should solve your issue.
I'm using AlarmDroid, which in its "advanced settings" has an option saying "rings max 10 minutes" (set by default), and just below a checkmark for "change to snooze-mode afterwards" (also enabled by default) -- so you could disable snooze mode, which would solve your issue (as the alarm would automatically turn off after 10 minutes).
Though I'd prefer to have snooze mode, plus an option to say "stop the alarm after xx if the user did not respond" -- some other app may have that, or it could be requested of the developer. The argument of "shomer shabbat" may be convincing for him (as pointing to a group of interested customers which might chose his Pro version for this feature).
EDIT: while searching for something else, I just stumbled upon Shabbos Alarm. The name alone seems to answer your question, doesn't it? It allows you to...
[...] set multiple alarms, choose which ringtones you want to ring, and set how long the alarms will ring for.
It works perfectly whether you call it Shabbos Alarm, Shabbat Alarm, Shabbis Alarm, or any other kind of Alarm for that matter :-)
Didn't try it out (yet), but thought you might be interested to hear about...