The problem is that average speed depends on time, not distance.
If you go the first 5km distance at 20km/h and the other 5km at 40km/h your total time is 22.5 minutes for an average speed of 26.67km/h (not 30km/h as you might have thought, see wikipedia on the harmonic mean for an extensive explanation).
Drag also increases with velocity squared, so all that precious potential energy you’ve gained by riding up the mountain will be lost to drag at high speed long before you’ve reached the foot of the mountain.
So generally it’s more important to be fast on the uphill parts. In the above example, increasing your uphill speed by 10% (e.g. by losing body weight and getting a lighter bicycle or by increasing your power output by 10%) would improve your overall time and average speed by 6.5%. Increasing the downhill speed by 10% would only improve the overall time by 3%. To increase the downhill speed by 10% would also require more than a 10% increase in power output.
That being said, even at 20km/h you’d still have considerable aerodynamic losses, so improving aerodynamics is always a good idea.