For a broad spectrum of riding my answer is Yes. Here's my logic.
There are 5 major principals to consider in no particular order. Air resistance, drag, weight, rolling resistance (centrifugal or rotating mass) and Friction.
In all of those 5 principals, the single speed has an advantage. So much so, that you'd quickly find that you need a bigger gear ratio to maintain the same optimum cadence, thus maintaining a higher speed given that same cadence.
The breakdown. Geared bikes carry more weight. They also create more air resistance and drag. The derailer carries more weight, air resistance and drag. The derailer also creates more friction from the pullie cogs themselves and the action of the derailer adding the tensions involved. Granted, a single speed drivetrain also has tension and friction, but only over two points of contact, not four. The friction and tension over those extra two points of contact while small, in combination with other elements, becomes significant. Rotating mass plays it's part as well. The mass of the cassettes, the larger hubs and it's internal parts, the derailer pullies, all of it, adds rotating resistance. The chain being longer and heavier also adds weight and rotating mass thus creating resistance. The hubs, cassettes, derailers (front and rear) and chainrings and chain length, all create extra points air resistance and drag.
Add all that up... to a lot of deficits. Having gone from 22 speed to 1X 7,8,9 to SS to Fixed and back to SS. Here's what I've found:
My Commute times drop by 7 minutes on my 16 mile commute and 3 minutes on my 6 mile commute going from geared to SS. Why? Because for the same cadence I'm in a bigger gear ratio so my average speed is higher overall. On the hills I don't have a lower gear ratio to drop to, so I accelerate to the ascent and hold my cadence as long as possible. This does two things, I start the ascent at a higher speed and cadence and it forces me to power on to maintain the cadence. Without the lower gear I find I crest the hill faster as long as the grade isn't such that I fall out of my personal powerband. With gears I drop to lower ratios, so while it was easier to pedal up the hill, I was also doing so at a slower pace, much slower in fact. If you're not going up mountain hills constantly why would you want to stay on the hill any longer than you have to? The faster you get up the hill, the faster you get off the hill! I find I save more energy this way as well as saving time. Hills add the most time to a route. If I spend 20 second's on a hill SS and 30 seconds geared? I mean I could power up the hill geared but it would still be a lower ratio than the SS because of all of the power sapping deficits. I'd still get up the hill faster!!! And then I'd accelerate down hill. Granted if the hill is really long I might be able to catch myself.... Maybe. Sometimes. Most times no. I've tried. I have to work hard to do it though, meanwhile I'm resting on the descent on SS...10 seconds ahead.
But overall? Same cadence and higher gear ratio equals higher average speed. Descent speeds on average are more than nullified by ascent speeds and flats. Again this is as long as your hill grades aren't too steep. Then of course you're going to pass me by. It's also why I don't recommend Fixed for long commutes with hills, you don't get to rest and your decent speeds are drastically reduced unless you feel comfortable unclipping or unstrapping or skidding a lot.
And last but not least, Single Speeds make you a much stronger rider, physically. I'm always amazed at how much stronger I feel going back to geared, especially when climbing. You've just built up so much power.
Disagree? Let me know, why.