We have the same issue with our Wileyfox Swift running stock Android 7/Nougat.
The default/AOSP messaging app plays the received alert through the earpiece during a call, which with our chosen tone was severely deafening let alone annoying.
It seems the default/AOSP messaging app somehow gets around or ignores the "Override Do Not Disturb" setting as shown in the attached screenshot below.
Depending on your firmware the solution may be to simply to toggle the alert options for the app (which has already been kindly pointed out).
If on the other hand this still does not work - like in our case - then another solution is to disable the AOSP Messaging app and install another one.
If you like to keep things standard then Google Messages is worth a try:-
Hope this helps.
(Tap to enlarge)