The trend towards compact frames goes back to the early 90s and an Englishmen named Mike Burrows. Burrows helped design several time trial frames which featured a radically lowered (but not sloping) top tube and a very long seat post. During this time most time trial bikes had top tubes that sloped from the seat DOWN to the head tube (the opposite of what we're now used to). Burrows thinking was that it allowed the rider to get into a lower position, a horizontal top tube presented less drag and that a reduced head tube would also result in less drag. Chris Boardman, one of the top time trialists at the time, rode a Burrows designed Cougar time trial bike and helped establish the look.
Soon Burrows expanded the compact design to road bikes but raised the head tube so the rider could be in the right position for road riding (versus time trialing). This created the now common upward slope in the top tube. He also thought that the compact sloping design would allow for fewer frame sizes being needed which would greatly simplify manufacturing and inventory. Adjustments would only need to be made by swapping out stems and seat posts. Burrows got Giant to embrace the concept in 1995.
Giant took over sponsorship of the Spanish ONCE team from Look in 1999 (might have been 2000?) The bikes were aluminum models and while very light they were also pretty harsh riding and the rear triangles were extremely short (the back of the seat tube was dented to allow clearance for the rear tire!) which made for poor descending.
The bikes sold in shops featured adjustable stems and airfoil shaped seatposts. While the number of frames dealers had to stock went down the shops were required to stock different lengths of seatposts as the airfoil shape severely limited the range of adjustment. This was almost worse than having to stock multiple frame sizes (I worked at a Giant dealership at the time) Also the adjustable stems were too flexible and heavy for use in pro racing and fiddly on the consumer side. Finally, three sizes turned out not to be enough and so Giant had to expand to offer 6 sizes which pretty much defeated one of the main selling points.
Reality is it's largely a fashion statement. For every pro there is a con. A sloping tube may make for a stiffer front triangle but a sloping top tube also adds more drag. Shorter seat stays make for a stiffer rear triangle which can make a bike more responsive in sprints but also doesn't absorb shock as well. A shorter seat tube is lighter but a longer seatpost is not.