How did coaches/scientists come up with specific interval training programs like X sets of Y intervals at Z heart rate/power, lasting T seconds? How are these numbers produced? Are these trial and error, is there good science behind them?
There are a number of possibilities but I can't say definitively. I'd guess it's grown from one or a combination of the following, in no particular order:
i. Consideration of event demands.
Coaches often create intervals which replicate the demands of a rider's goal events. e.g. if you are racing an event with a lot of short climbs which are often attacked, then some hard efforts of around that duration might be worth considering. Or in the case of track racing, it might be a certain number of laps of the track, with sufficient recovery to go again. And of course if you are doing time trials, then efforts that replicate time trial pace and duration are common.
ii. Published studies involving interval training.
There have been over the years a number of published studies which involved various forms of interval training. Many of these have been shown to be physiologically beneficial in some ways, and so these may have formed the basis of many such interval workout designs (e.g. Tabata). As has been commented upon, whilst many such workouts have demonstrated benefit in laboratory study, they are by no means necessarily optimal for any given individual, or for the specific requirements of that rider.
I should add that many of the published studies referred to were not primarily about identifying whether or how much an interval session was beneficial but rather it may have been about answering a different question, such as identifying limits of human performance, or in research on physiological (e.g. hormonal, blood lactate, VO2 etc) or psychological responses to particular types of effort, or indeed simply as a way to test models of human performance, or to assess the validity of particular testing methodologies, or what happens when other factors are controlled (e.g. environmental conditions). Indeed many studies are not performed on trained cyclists but are primarily about assessing health impacts of such workouts on untrained individuals or for those with particular medical conditions. But along the way interval prescriptions and their impact were measured and noted.
iii. The terrain and riding environment people had available.
e.g. some people had a local climb which took about 5-minutes to climb, and so they ended up doing hill repeats of around 5-minutes. Or they had a section of flat road which enabled ~20-min of hard steady state effort before they had to stop for some reason (e.g. edge of town, traffic lights, end of road). Turn around and do it again and voila, 2x20-min intervals are born. Or they are time limited and/or on an indoor trainer and so they are looking for ways to make the session useful as well as sufficiently mentally stimulating. Lots of interval sets are designed to fit within a given time frame.
iv. Mental considerations
Riders need variable challenges in their training diet to keep it mentally stimulating, fun even. A way to reduce the drudge of banging out the same old miles every day. Intervals are perfect for this, while admittedly for others they'd rather scratch their eyeballs out. So coaches design training challenges as way to balance the psychological needs of a rider with their physiological development. Training needs to be sufficiently enjoyable, else you are less likely to do it.
v. A way to track improvement
When you do a set of intervals on a regular basis it makes for a good way to measure improvements in performance, and also when performance is no longer improving, it's also a sign a change in routine is perhaps warranted.
vi. Folklore. It worked for someone else, so why not try it on others?
A "what we always did" or "they did it and it worked" kind of approach. A mythical secret training sauce. Again by no means optimal nor necessarily specific to an individual's particular needs, but as I'll mention later, not always as bad as some might think. "We did these efforts and through trial and error discovered we needed 10 minutes recovery and could do 5-7 of them at most. And our riders were flying 5 weeks later."
vii. Understanding the key physiological capabilities, capacities and development needs of a given rider, and formulating training workouts that are specifically suitable for them.
This is where one forms a deeper understanding of what sort of training efforts are most suitable for a given individual's physiological development. This is in part informed as much by the published research as it is by measurement and monitoring of the specific responses an individual has to such training, especially via the use of a power meter.
There are only a handful of physiological capacities one needs to be concerned with and the balance of these attributes helps guide what sort of workouts are suitable.
The reality is that the impact of training at different intensities on stimulating beneficial changes to these various physiological attributes is on a continuum. What this means is that most intervals sets provide a development stimulus across the range of physiological capability. e.g. just because you are doing short hard intervals doesn't mean they won't also have an impact on longer range aerobic capabilities. It's really about targeting the primary adaptations you seek to improve while also recognising the impacts also spill over to other areas.
And this is why, despite some level of folklore and ad-hockery to how such things arose, the physiological responses to such efforts still, by and large, work.
Finally, while intervals typically are mostly about physiological development, there are of course other factors to consider when it comes to a rider's improvement: skills, mental approach, recovery, team work, strategy, tactics, and often interval training can also incorporate these elements (e.g. sprint training with a mate, team pursuit style efforts and so on).
There are thousands of published scientific studies involving various uses of high intensity interval training protocols. There are also many studies comparing HIIT with predominantly threshold level aerobic and/or steady state training.
Listed below are some studies comparing HIIT protocols. This is by no means an exhaustive list of references.