I'm going to attempt to approach the question a different way than the other two answers. This issue has been on my mind for a bit. Note that the answer refers to my experience in the US a lot. I do so only because I live in the US; the things I discuss likely go on elsewhere, but I'm not as aware of dynamics in other parts of the world. Basically, certain aspects of traditional road racing likely inhibit the experience of many amateur riders. However, there are other bicycle-based experiences that have competitive dynamics and are friendlier and more enjoyable for amateurs.
Running as a counterexample to cycling
In running, there seems to be a lot of room to let the primary competition be yourself. In other words, while you can focus on your placing relative to the competition, there is a lot of room to just focus on bettering your previous time. Running races are over a set distance, e.g. a marathon is 26.2 miles, a 5k run is 5 kilometers. While environmental conditions and elevation obviously influence your pace on a given day, as can race dynamics, there's an objective measure for your speed, e.g. running a sub 2-hour marathon in 'real world' conditions is currently just out of reach for the very top male marathoners. This means that you don't have to be "ridiculously fit" to derive enjoyment from the sport. You just have to better your previous self, or your expected self, or something similar.
Competitive road cycling may be too optimized for elites
When the original poster said "competitive cycling", they seem to have meant road racing. Aerodynamics shapes the tactics of road racing much more heavily than running (although NB, Elihud Kipchoge's benefitted from the draft of a peloton in his sub 2 hour marathon distance run). That is, in road racing, the entire peloton might well finish at the same time, i.e. the race ends in a bunch sprint. It is much more difficult for a breakaway to succeed, unless the rider(s) is/are that much stronger than the field or other team dynamics are at play. Thus, we don't really pay much attention to objective finishing times in regular, competitive road racing.
In principle, one could imagine a sub-discipline of road racing where you pay more attention to objective times. That would be time trials. Unfortunately, you can buy a lot of speed with aerodynamic equipment, and time trials have always been a sub-discipline of road racing. That said, in the UK, I believe there is a regular amateur time trial circuit, and there may be in other countries.
Because an objective standard for finishing times isn't as viable as in running, road races seem to devolve towards just who wins or places top 10. All races have a winner takes most or winner takes all sort of dynamic going with regards to attention, financial rewards, and the like. In traditional road racing, I suspect that dynamic is amplified. I've heard it said that you have to train very hard just to suck in road racing. At the time of writing, there seems to be a strong consensus in the US that traditional road racing is dying off. I suspect that was also the consensus at the time the original question was asked.
I recall a Youtube video by Ross Tucker on talent identification and management. I forget which specific video. I believe he made the passing comment that a country can optimize its sports talent identification and management system to produce elite athletes, or to produce broad participation. He didn't give an example of what these optimizations look like, nor was there a lot of context. However, I'd posit that in traditional road racing, the talent identification and management system, as well as the rewards system, are all set up to maximize production of elite athletes. This comes at the possible expense of broad participation.
I lack the same depth of experience with running, but it could be that the sport as a whole has aimed more at a balance between broad participation and producing elites.
There are alternatives off-road
There are a growing number of alternatives to traditional road racing. In the US and likely in other countries, there is an emerging gravel racing scene. At least right now, the races are very participant-driven rather than being elite-driven races or categorized like traditional road races. Gravel races may let the elites, if there are any present, start first. However, for the most part, it's a full mass start, like a marathon.
The terrain, lower speed, and broader spectrum of ability levels all do seem to attenuate the aerodynamics-driven race dynamics that are in play in road racing. That is, you don't expect one big bunch sprint, and riders typically arrive separated by quite some time. Even though tracking your finishing time doesn't seem to be a trend, there does seem to be a big focus on participation by a broad range of riders. Many people do it for the intrinsic challenge.
It is likely that we will see more professional gravel racers. It's possible that if the whole sport gets too professionalized and elite-driven, it could sour competitive dynamics for the rest of us. However, right now, this is speculative.
In the US, Dirty Kanza is probably the most well-known such race, but races of a similar level in the US might include Rebecca's Private Idaho, Mid South, and SBT Gravel just to name a few. Smaller-scale rides that have strong regional competition also exist; I posit that the now-closed Almanzo ride in Minnesota and the ongoing Trans Iowa ride are smaller-scale but well-known gravel races (NB, Almanzo spawned two successors in 2020, the Heywood and the Spring Valley 100, but these are not ongoing this year due to the COVID-19 epidemic). It is likely that there are at least some smaller events in your state or a neighboring state.
There are alternatives on road also
There are similar participant-driven rides that are on tarmac, or are primarily on tarmac. Bicycle touring, audax cycling, and similar disciplines might be one example of very long-distance road riding. I am not as familiar with this type of riding, but proponents would say that this has been going on much longer than gravel. In the US, the Race Across America could be the most well-known of these, although the distance might be too formidable for many amateurs. Multiple-day road rides might be more acceptable to a broader crowd.
There are also century rides (100 miles) and Gran Fondos. It is likely that at least some participants here will ride these in an informal competition. If you want a road racing-like experience, I suspect you might be able to get it at many of these events. One of my most memorable rides last year was the Tour de Tonka in Minnetonka, Minnesota. I completed it in 4 hours 33 minutes despite getting dropped from the lead group. Yes, there was a lead group. No, I'm not in that sort of shape. However, because of aerodynamics, I was able to hang on for the majority of the race. After I dropped off, I was able to regroup with smaller groups of riders who were closer to my ability level.
If you are not up for a century ride, many centuries organize shorter distance rides. For example the Tour de Tonka and the Jesse James bike tour in Northfield, MN, organized multiple distances from about 20 to about 80 miles. Because the distances involved are often somewhat objective, you could aim to better your time at a certain distance. However, there are aerodynamic effects involved; for example, there is no way I could have completed 100 miles solo in 4 hours 33 minutes. (Also, the distances will vary from year to year due to road closures; the 2019 Tour de Tonka century was actually 98.15 miles.)
