I think I understand the derailleur mechanism pretty well, yet I still find it surprising how the chain catches on the new cog so reliably. Is there something I'm missing? Is it just with the proper derailleur alignment, the chain will inevitably catch?
A great deal of engineering has gone into making shifting smooth and predictable. Up until ~1984, shifting was done by feel and sound: the shift lever moved continuously through its range, and you moved it until it caught the adjacent gear. Different brands had different actions, and you had to develop some technique.
In 1984, Shimano rolled out "indexed shifting" (there had been previous attempts, but this time it really took). Here, the shift lever clicked into specific positions; the spacing between gears on the freewheel was standardized (before, the spacing varied within a freewheel); the shift cable was pre-stretched so the action wouldn't change; and so on. There was a lot of skepticism that indexed shifting would stay reliable in the long run, but it did, and took over.
Since then, there have been additional refinements. Individual gears on a cassette have "ramps" that help lift the chain from smaller gears. Gears are aligned relative to each other to ensure good uptake. Chainrings have pins to help lift as well.