Is there a verifiable reason for the ubiquity of flat bars on bikes these days?-- if so, what is it? It seems that 90% of the time I see people riding a newish bike that isn't a Dutch bike/roadster, they are holding their arms straight out in order to grab onto some flat piece of metal covered in rubber. Moreover, if it's a typical "hipster" bike (i.e. an older frame possibly with newer components), even if it has drop bars, the brake levers are not typical "drop-bar-style" brakes but are rather mounted horizontally along the bar tops, meaning that the hooks are more or less useless because you can't brake from them, and the drops are even more dangerous because it's just not possible (for me at least) to quickly move from the drops to the tops quickly while keeping myself stable (My Google-fu doesn't seem to be good enough to find a photo of this phenomenon online).
Caveat: This answer features rampant personal speculation and anecdotal evidence.
I personally the current dominance of flat bars in mainstream cycling is the result of two main events:
- The flood of cheap 10 speed knock-offs entered the market in the 70's and 80's.
- The explosion of MTB bikes in the 80's and 90's (which featured flat bars).
Event 1: Low end 10-speed market dominance
Chasing the popularity of road racing and riders, such as Eddie Merckx, the 70's and 80's featured a flood of cheap 10 speed road bike knock-offs that had the primary goal looking similar to the types of bikes that Eddy Merckx et al. were riding at the time. These bikes featured drop bars with poor ergonomics and frames that likely had too low a stack height for mainstream riders. Because these bikes did not fit most people very well, the general public was left with the impression drop bars were uncomfortable and should only be ridden in very aggressive positions.
Event 2: Explosion of MTB bikes in the 80's and 90's
The mountain bike explosion of the 80's and 90's further maligned drop bars. These bikes featured more upright riding positions due their off-road nature which most people found more comfortable. While early incarnations of mountain bikes featured drop bars (Figure 1), straight bars quickly became a fixed feature (likely for the improved leverage - although this is speculative on my part).
Figure 1. Charlie Cunningham riding an early mountain bike featuring drop bars.
As a result, many people associated comfortable with a bike featuring straight handle bars and as a result you essentially have historical contingency determining the current market dominance of flat bars. If you have a background in either evolution or genetics, this is can be view similar to how an an allele can become fixed in a population by random chance (i.e., genetic drift).
Finally, I find it fascinating that "dirt drops" have been making a come back in recent years. What is old, is new again... either that or we are still experiencing genetic drift!
Flat bars are easier to learn on than drop bars (unless the latter have interrupter brake levers). At this stage low speed manoeuvrability matters more than efficiency. Some people with small hands apparently have trouble braking firmly from the hoods and getting into the drops is tricky for a novice (I dismiss starting in the drops as I've never got on with it). Then people like what they're familiar with if they lack a pressing reason to switch. Drop bars - as you said - need to be right to be comfortable, and the riders who get even the most basic bike shop fitting (not good enough for road bikes really) are in the minority.
Some people (e.g. me) like the extra height of flat bars for seeing and being seen over cars.
Cruiser bars (apart from being a horrible angle for your wrists if the top tube is on the short side and/or you have broad shoulders) lead to odd handling on some frame geometries, and make it hard to stand up on the pedals. So they don't work for MTB use. And MTBs have sold well since they went mass market (including in BSO form as evidenced in a recent question here). This may have something to do with the forgiving geometry - including the hands position. Don't underestimate the value of a bike that doesn't need much fitting (just saddle height) to be adequate.
OPINION Drop bars should never have been popular.
There was a surge in popularity for the "ten-speed" in the 70s and 80s, and part of that look was drop bars. This artificially made drops seem like a common thing and a good idea.
As the MTB arrives in the 90s, flat bars re-exert themselves for the added width and leverage, returning drop bars to their less-common status, where they should have been.
Exacerbating this fall-from-cool is that road bikes are getting more slammed or aggressive, with higher seats and lower bars. Compare Fignon in the 1980s:
http://bikeretrogrouch.blogspot.co.nz/2013/10/changing-positions.html is a great read on the subject.
Personally, I am on the hoods most of the time, moving to the tops for a steep climb, and pretty much never on the drops - My thighs hit my stomach while pedalling which is horrid, plus the other crunched things complain.
Edit: after a couple years my belly no longer hits my thighs while on the drops, so I can stay there for a good 10 km if its windy.