Simple answer: Yes, but it's usually not noticeable and won't impact a device's ability to function as expected during it's lifetime.
Full Explanation: Apps CAN leave traces of information behind when uninstalled. These are usually empty folder and misc. log files, but can also include files created by the application but are assumed that the user would want to keep (screenshot folders come to mind). Although these hardly affect performance, they can eventually take up space and introduce unwanted clutter in your mounted storage volume. Purge with aplomb.
Another area of possible concern is the concept of "bit-rot", or "bit-decay". This is the decay of data stored in a solid-state format, unaltered, for long periods of time due to the dispersion of the electric charge used to maintain the data. An OS of a phone could be subject to this phenomenon over the course of many years (usually more than a decade) because the data of an OS rarely changes in the scenario you present (not updating and never re-installing). Though reinstall the OS will refresh the memory locations it's stored on, the process of bit-rot takes a long time on properly manufactured devices, and isn't likely to cause any sort of problems in the 2-3 year timeframe most individuals keep their phones and tablets nowadays.