It is mostly too late indeed, unless you are already a world class or at least national-level athlete in a different sport - like Primož Roglič who was a ski jumper. Human bodies need the right level of physical training during their growth, that means some reasonable training during childhood and especially during junior levels to get to world class levels. The competition is too large. You could still perhaps get to some very high levels if you have the right genetic predispositions, but the Olympics are very unlikely and professional cycling likely as well.
I absolutely support the recommendation of Adam Rice to join any local race and see how you compare with others. If you are really gifted, you would see that you can easily compete with most other people. Where I am from, there are many MTB amateur races for all age groups and abilities. The possibilities of road or track racing, on the other hand, are very limited. Your country or even your state may be different, see what is available in your area. For the start it really does not mater too much if it is road, track, cross-country or cyclocross.
You are looking for the signs of exceptional physical strength/endurance even if you see that your technical skills are holding you back. If you have the genes for excelent physical condition, some coach may even spot you at these races because you would start winning soon.