Is there any evidence to suggest that cycling adversely affects male fertility?
Yes, but only in fairly extreme cases.
According to this article, cyclists who regularly cover more than 186 miles (300km) per week are likely to have fertility problems.
If you're not doing that kind of mileage then I don't think you've got too much to worry about.
I've just found this essay - Great balls of fire and the vicious cycle: A study of the effects of cycling on male fertility
According to this essay there is some evidence that cycling long distances can cause fertility problems in men, however it is far from conclusive. Nerve entrapment, trauma or increased scrotal temperature are potential risks for the male cyclist.
Most scientific studies on cycling and urogenital problems are written with clinicians in mind, to make them aware of possible symptoms that they will encounter. Often these studies get summarized into review articles. One such article (Leibovitch and Mor, 2005), reviewed 62 relevant studies. They say:
When examined individually, many of these studies have limitations (small number of study subjects, no followup, etc.), but when taken as a whole, they are pretty conclusive that an association exists between cycling and urogenital problems.
The only study that I could find that was population-based, i.e., included men with a range of cycling durations, was Marceau et al., 2001. Their data on 1,709 men come from the Massachusetts Male Aging Study. After controlling for potentially confounding factors ("age, energy expenditure, smoking, depression and chronic illness"), they found that men who cycled more than 3 hours per week were 72% more likely to have erectile disfunction than those who cycled less than 3 hours per week. They are cautious, however (emphasis mine):
The last sentence is key. Granted, there are drawbacks to population-based studies like this. One major problem is that the data are self-reported. Another drawback is they their sample probably does not include elite or high-level sport cyclists, who might have better equipment or better technique.
Does this mean that your experience may vary? Absolutely.