If your situation is improved by
Don't keep activities, this is a strong indication that you are running poorly developed applications. The whole point of preserving applications in the background is to make full use of memory and reduce latency. If this increases latency, this means your applications are trying to do a lot of work in the background.
I would not expect battery life to be reduced by
Don't keep activities, if the startup time of your applications is lower than the latency of keeping them around and switching. I would actually expect your battery life to improve by killing resource hogs immediately.
If you do not mind spending time diagnosing this, you should work on finding the culprit application(s). This may involve a substantial amount of testing combinations of applications. It may be that a particular combination of applications is creating contention for a resource that is handled differently on the Nexus 5X than on other devices. One application alone may not be the problem; it may be a bad interaction among a combination of applications that causes a major issue.
A less time intensive solution is probably a service like Greenify, described in How to prevent apps and processes from launching in background and slowing my device.
I have a Nexus 5X, and I do not any issues with interface latency. I also do not use many applications that are not stock.
In the past, I have had significant issues with latency resulting from filesystem TRIM on a Galaxy Nexus. With a recent explanation of when fstrim is scheduled to occur, I have modified my charging behavior to leave the phone plugged in and unused long enough for fstrim to run, and this helps quite a bit. I don't know, however, if the fstrim behavior for Marshmallow is the same.