As others have indicated, it depends on both your bike and the type of riding you're doing.
If you're just tooling around town (get it, "tooling"?) with no particular destination or schedule, you can probably get away with not much more than a set of tire levers. A pump is handy too so that you don't have to walk your bike to a gas station if you get a flat.
If you're commuting, you may want a bit more. I carry the Park Tools MT-1 in my saddlebag along with a tube and a set of tire levers. It has enough to get me from A to B with the ability to solve minor problems inbetween, but it's not bulky and it's cheap enough that if somebody steals my saddle bag (which has happened to me three or four times) I'm not too distraught. My commuter is a single-speed so I also carry Surly's Jethro Tule for adjusting the rear wheel.
On longer rides, you'll want even more. Like Drew Stephens, I'm a fan of the Crank Brothers Multi-17. You can do almost anything you're likely to need to do with this thing, albeit somewhat awkwardly in a few cases, but that's just the nature of multi-tools.
The above advice probably applies to a short, weekend tour as well. However, if you're going to be straying too far from civilization or going longer than a night or two, you may want to consider carrying something beyond a multi-tool. There's really no substitute for a chain whip when you're stuck in the middle of nowhere. Same goes for crank and bottom bracket tools.