There are several factors, which I'll address (pun intended) in no particular order.
RAM is expensive
Sure, memory chips may be cheap, but that's not the only (or even the main) cost. Alongside the RAM itself you need extra buses, power lines, bigger memory controllers, heatsinks, &c. The RAM also takes physical space on-chip. For oomph-per-dollar, at the moment, you're better off increasing cache sizes rather than memory sizes.
Also, don't forget how power-hungry RAM is. Doubling the RAM capacity means doubling its power budget, which means either your battery life gets even worse, or you need bigger batteries. Now, you could say that the extra RAM could be powered down until it's needed, but (a) don't forget that Android currently tries to keep all the RAM full to save CPU cycles, so the software changes to support that would be very expensive, and (b) that's just giving app developers another way to screw up and run your battery down.
So, it's not as cheap as all that, and you don't want to add more RAM until you can really use it, which you can't, because of
32-bit processors
Remember how desktops and servers only started having lots of RAM when 64-bit processors became popular? That's because 32-bit processors can only have 4 GB of address space. That has to include all the RAM you want to access, plus any memory-mapped buffers for accessing other hardware and storage. By the time you count everything, it effectively limits you to accessing about 3GB of RAM.
With LPAE (the ARM equivalent of Intel's PAE) you can make that a per-process limit rather than a whole-system limit. That is, you could have 8 GB of RAM in your device, and each process could use up to 3GB of it. But that's not as useful for a phone as it is for a server, because of
Single-tasking
Unlike iOS, Android lets apps run background services, but realistically, they're not going to be doing memory-intensive operations. You don't need 2 GB of RAM to sync your email and contacts, and for more intensive background operations, the trend has been to put them in "the cloud", letting big servers handle the heavy lifting and just send the results to your screen. If you think of a smartphone and tablet as just a thin client connected to Internet services, there's no way you'd even need 1 GB of RAM. RAM demands have only grown past that because of increasing screen resolutions and the need to supply large textures to games.
You mentioned potential for multitasking, but it's a tiny minority of users would would ever be swapping between two memory-hungry apps on a tablet or phone. Maybe a paint program and a 3D modelling app? This may be something that changes if manufacturers start bringing Android to desktops or more powerful portables, but right now RAM is not the only thing stopping people using their devices that way. Even if you are multitasking, you have
Fast non-volatile memory
Android devices have an extra advantage that means they don't need as much RAM as desktops or servers: fast non-volatile storage. You mentioned swapping to disk in your question, so don't forget that Windows has been doing this with ReadyBoost, which uses a flash drive as virtual memory (the Windows equivalent of swap space). Using flash memory as swap is actually pretty fast, especially when it's internal memory (so the system integrator can tune everything for its latency and bandwidth) rather than an arbitrary SD card.
Android already lets apps pack up their GUI state into special storage before they go away. This lets background processes take up less memory, and provides a path for Android to move that state to non-volatile memory. I'd expect to see this mechanism used a lot more cleverly in software to make the most of the RAM you already have, before manufacturers start pushing the 32-bit limit.
To summarise: there's no reason Android devices can't have more memory, but adding more isn't just a question of going to PC World and buying a stick of DDR. At the same time, most users won't see any benefit from the extra memory; the way Android devices are used and the way Android apps are written would have to change before they would.
If you want to replace your home server with a cheap, power-efficient ARM computer, there's no need to look to Android for that. You can get small ARM servers that run GNU/Linux, and even some mainstream NASes are easy to install a GNU/Linux distro on. You might even be able to repurpose an Android device for this (for example, Ubuntu 13.04 is supposed to run on a Nexus 7).
