I have gone through two major upgrades to my road biking experience, and I felt the need to check if they made any real-life difference. I figured it might be useful to share. Noting that while I tried to go about this with some level of science, there are obviously lots of faults in the methodology.
Equipment
Old: Giant Defy Endurance road bike with standard wheels
Aero bike: SRT Aero 1.19 with Bontrager Affinity TLR wheels. It's not the most aero bike out there, but at least this bike has integrated cabling
Aero wheels: SRT as above but with Lun Routte SE wheels (45/55mm)
Methodology
Take average power meter readings from all rides on a flattish Strava segment as well as log the time for the segment. This includes data over a period of about 3 years and conditions that vary significantly - wind, rain, drafting, smaller gear changes (like helmet / jersey), rider weight fluctuations, potentially some improvement in posture, etc. Throw out any clear outliers like punctures or other stops.
The segment I chose is https://www.strava.com/segments/9082414 - it's one of the flatter ones I ride often enough to have a good amount of data.
Run a trendline through the data in Excel to see if there remains a meaningful difference nonetheless. Try and quantify this difference for the two major upgrades at different power points. I stuck with exponential trendlines as that is closer to how it should increase with speed and therefore power (in a perfect world).
Results
The scatter plot is below:
What is this telling me?
1. There is significant variation at the same power point, even with the same gear (you can see this in the relatively low R-squared values). This points to the fact that obviously it's not just the gear that determines the speed. Knowing the conditions much better in the Aero Wheel dataset (because they are recent), I believe the most significant contributor by far is wind.
2. Following this point, I saw some quite significant variations at the same power point on the same day, where you would have expected wind conditions to remain very similar. I chalk this down to the inherent gustiness that happens next to the mountain, you can be lucky or unlucky over the 4/5 min.
3. The Aero bike advantage is significant enough that it matters. Weird that the trendlines get closer together at higher power, but I am sure that's just a limitation of the amount of data.
4. The aero wheels are a much bigger advantage, and diverge a bit more like you would expect. Just because I have been excited to test the wheels a bit more, I have probably gone out on more windy days than I would normally have which means the difference is potentially understated
What do my trendlines predict at different average power?
Watts
Old
Aero bike
Lun
100
06:09:39
06:00:41
05:50:14
150
05:43:06
05:34:57
05:21:13
200
05:18:28
05:11:04
04:54:36
250
04:55:36
04:48:53
04:30:11
300
04:34:23
04:28:16
04:07:48
325
04:24:21
04:18:32
03:57:19
350
04:14:41
04:09:08
03:47:16
At 300W I am already seeing a potential 26 second gain over a ~4min run - that is a crazy improvement! I have done 325W for this segment before (on the old wheels), and it looks like just the wheel upgrade would have gotten me below 4 mins on that day.
Limitations
There are lots of issues with this analysis, as I did not plan on doing this when I started "collecting" data:
Drafting - some of these data points will include drafting, but I estimate that to be less than 5 of the total, and definitely none in the Aero Wheels set.
Wind - Hoping that with enough data points this starts canceling out, but it has such a large effect that we won't really know without tracking it, and tracking it is very hard due to the gustiness next to the mountain.
Rain - this also makes a bigger different than you think, even just a wet road makes a difference as some of your energy goes into throwing those drops into the air.
Weight fluctuations - While I tried to look for a flattish route, there is still a small amount of elevation change. My weight has fluctuated between 82 and 89 kg for most of these data points. The newer data is around 85 kg.
Other gains - Over the time period I got a better helmet, better jersey, moved to tubeless GP5000 from tubed GP4000, and have probably improved my position a little bit. Of these the position should be the most impactful, but I tried keeping it pretty standard for the newer data (riding in the drops).
Power plot - it does make a difference how smooth your power output is. It seemed like smoother power output gave me faster results.
Statistical analysis - there is some more statistical analysis I could do, but I've gotten enough out of this already so will stop here.
Conclusion (TL:DR)
Upgrades do make a measurable difference to a normal bike rider (it's not about the bike, but it is a little). You can buy speed!
Upgrading wheels makes a much bigger difference than upgrading the rest of your bike.
I can finally justify buying that power meter some years ago
Thanks for coming to my TED talk.