About functionality of road bike kit
For distance riding there a number of functional advantages to road bike style clothing (road kit), these include:
- High quality chamois reduces the chance of saddle sores;
- Tight fit reduces aerodynamic drag and effort at higher speeds;
- Sweat wicking properties keep you more comfortable;
- Tight fitting makes it easy to layer clothing to match environmental conditions;
- Tight fit keeps clothing from getting caught when moving around on the bike:
- This is why XC racers wear tight kit as it keeps your shorts from getting caught on the saddle when you shift your weight behind a high saddle (i.e., riding drops or braking hard); and
- In Cyclocross (CX) this makes mounting and dismounting much easier, as you literally step over the saddle and onto the pedals in one motion when mounting on the run.
Apart from (2) it is reasonably easy to find varying degrees of these features in other "baggy" style bicycle clothing. If you are riding a lot of gravel then your average speed will be lower so (2) won't matter as much but (5) could still be useful.
About Clothing Conventions
As others have commented on, if you like it, then that is all that matters. If a roadie squawks tell them, "Life's hard, get a helmet!"
But in all seriousness the only places I would consider baggy wearing clothing over bike gear (to look more "normal") is areas of the world that have strong cultural identities that may clash with this type of clothing. Sometimes this can lead to a short-circuit in the brain, causing behavioural problems.
I can think of some places in every continent of the world where this could be an issue.