The people at Audax UK have given this some thought. They regard the gold standard as counting contours on an Ordnance Survey map (for those not in the UK, the output of our national mapping agency can be expensive, but is very good). This is rather tricky in areas with lots of features, such as towns, and is tedious at the best of times. It works best for sustained climbs in open country -- which is what they mainly want it for. So they've tried to find an automated system that's just as good. For track logs they recommend http://www.bikehike.co.uk (UK) and http://www.ridewithgps.com (overseas) which are both free. Bikehike works for routes as well (in the UK). As my original question was about building routes, I've tried a few tools for comparison to my own logs.
Test 1: 20km of undulating road and bridleways with around 230m of climb
I've recreated a ride I did today using Strava route builder and bikehike
Strava needs to have 1km and 10m added to account for different start/end points (I don't start Strava stuff from home, even with privacy zones).
Both are a good match to my GPS log (which may actually use the barometer):
Method Distance/km Total climb/m
IPBike 21.513 231
Strava track 21.4 233 (Strava's interpretation of IPbike data)
Strava route 20.8 207 (Recreated online)
Bikehike 21.44 204
OS Map (not measured) 220 (Counting contours with a magnifying glass)
These all seem to be within a reasonable margin of error (bikehike was 12% less than my GPS, the biggest difference). Note that this route wasn't hilly, and didn't run in any steep-sided valleys, so lateral GPS errors will only have had a small effect.
I counted 5m contour lines on the 1:25000 OS map, using a magnifying glass. I'm surprised that at the first attempt I got so close to my measured data. In some places the road almost follows the contours and in other places the contours are obscured by other features.
Unfortunately Google doesn't understand UK rights of way very well, and can only get a small fraction of the off-road sections, so my comparison can't be extended to include that.
Test 2: 67.5km of road with around 470m of climb
I applied the same test to a longer road ride (but not one I plan to publish). Much of the ride is quite flat, with a couple of decent climbs. I do this ride every few weeks so have some average data. We have:
Method Distance/km Total climb/m
IPBike 67.51 464
IPBike average 67.63 473.57 } of seven rides
IPBike StDev 0.252 20.743 } over the same route
Strava track 67.51 445 (Strava's interpretation of IPbike data)
Strava route 67.6 557 (Recreated online)
Bikehike route 67.29 445
Google route 67.5 285
The above are from 2017; things may have changed.
Added in 2020 now Strava Route Builder is paid:
Komoot 67.7 380
RideWithGPS 67.7 407
Taking the average measured data as a reference, all the horizontal distances are within 0.5%.
Vertically, bikehike is a good match to my data (and recall that AUK reckon it's a good match to counting contours). Strava route builder over-estimates by 18%, but Google under-estimates by 40%.
I don't have all the OS maps for this route at a sufficient scale, neither do I have the patience (and the contours might well be omitted in the urban sections).
Segments are created by users, and appear to be based on the originator's tracklog. Some are clearly prone to the error reported in Criggie's answer, as they have vertical jumps, as demonstrated by one I rode recently. In this case the error causes the segment to be called a Cat3 climb. It's nowhere near.
When building a route:
- Strava might be a little generous -- good for bragging rights and gives a margin for error.
- Google can seriously underestimate. I'd say that this is enough to be dangerous -- if you're pushing yourself according to Google and it's 40% under you're going to really struggle.
- Bikehike seems good, but is mainly for UK riders
- Horizontal is much easier than vertical