All computers have a special piece of hardware called the clock (more fully, the real-time clock or RTC). The clock is connected to the CPU, but it runs even when the computer is completely powered down. On a desktop computer, a battery on the motherboard keeps the clock running even when it's unplugged from the mains. Mobile devices tend to have a large capacitor (which is a bit like a rechargeable battery), so even if you pull the main battery out, the clock can continue to keep time for some hours.
At regular intervals, the clock sends a signal to the CPU called an interrupt. It tells the CPU to interrupt whatever it's doing to process this signal. This is the same mechanism used by other parts of the computer (such as the touch screen) to send input to the CPU. The rate of the signals can be controlled by the kernel: it might be as slow as once per second, or as fast as a thousand times per second. This sounds like a lot of signals, but don't forget that today's computers take a fraction of a nanosecond to do a single operation, so even a 1 kHz clock only interrupts every few million operations.
Don't confuse the real-time clock with the CPU clock and the CPU's clock rate, which are typically a few GHz. The CPU clock is not a timekeeping device: it sets the number of operations per second the CPU can perform, and is a big factor in how much power the CPU draws.
On most CPUs, the kernel can decrease the CPU clock rate, turn off some CPU cores, and maybe even power down some caches, all to reduce power consumption. This is what happens when your Android device goes to sleep, but the exact changes are different from one phone to another. The CPU is still running, just more slowly, and drawing less power. As Izzy points out, if the phone uses ARM's big.LITTLE system, it might even be a completely different CPU core that runs in sleep mode: a slower one tuned for low power consumption.
Consider, then, that the RTC is still running in sleep mode, and the CPU is still receiving and processing clock interrupts. This is still necessary for the CPU to keep track of different processes and allocate time between them fairly. One of the processes is the system's alarm manager. This has a sorted list of all the apps on the system that have registered an alarm. The alarm manager doesn't run continuously: rather, it tells the kernel when the first alarm is, and the interrupt handler for the clock interrupts is responsible for checking the time and waking up the alarm manager when appropriate.
This way, the CPU only needs a few instructions per clock to compare the current time with the time of the next wake-up.
In addition, the alarm manager has its own optimizations to keep the device asleep as much as possible. When several apps want to run at intervals (e.g. to check email every 15 minutes), the alarm manager won't run them individually at the exact times they asked for, waking up the device several times over that 15 minute window. Instead, it will run all of them at the same time, so the device only needs to wake up once.
On top of that, many alarms won't run at all if the device is asleep. For example, if an app wants to run every hour to change your home screen background, there's no point in doing that while the device is asleep, because you can't see it. It will tell the alarm manager that it doesn't need to wake up the device to run, so the alarm manager will ignore that alarm when telling the kernel when it needs to run. That way, if the device is asleep when the time comes, the alarm will only happen when the device next wakes up.