If you're always close enough to call help when you have serious mechanical trouble, then a racing bike will make you lighter and faster, but the speed advantages that racers realize often come from wrapping themselves around uncomfortably small frames on tires that require more focus on descents with components that have a lower mean time to failure for their price.
There's nothing wrong with taking a day to do an English century as long as you enjoy it. In my late 20's, my extended tour rate for alpine topography was about 80 miles (128 km) a day on a half-loaded (no tent) touring bike with occasional days in the 100-120 mile (160-200 km) range.
I would've gone faster on an unloaded racing bike, but I wouldn't have been able to choose my own course for fear of being stranded without spare parts in foreign lands. And every kilo in weight I add to the bike is a kilo less I weigh at the end of the summer, and a touring bike is useful during the off season.
If you can afford it, try a fast bike so you can talk trash with your buddies on short aggressive rides, but for comfortable riding and peace of mind, a little extra weight for durable components, a comfortable saddle, a fitting frame, and spares is worth it.
PS: I may be overboard. For my commute, I carry panniers, a laptop, tools and raingear that weigh in at 49.6 lbs (22.5 kg) so my frame material and tire width aren't what slows me down.