Yes, geometry matters. If you're happy with your bike, then you don't need to worry about it.
There are a lot of different cycling disciplines and sub-disciplines: for example, within mountain biking, there's cross-country and downhill and others. Within road racing, there's criterium, point-to-point, time trial, and so on. And there are bikes with geometries, material selections, wheels, tires, and other components that are optimized for each of these disciplines. Sometimes the differences are tiny, and the sort of thing that very few need concern themselves with.
Consider, for example, that a bike with very sharp handling will be beneficial in some situations (like criteriums), but if you don't need that, you're just going to expend more energy keeping your bike going in a straight line, which at the end of a long ride would leave you more fatigued. Or that some touring bikes are designed so that they handle better when panniers are mounted on the forks. Or that using a racing bike for touring may leave you with your heels hitting your rear panniers because the chainstays aren't long enough.
Bike fit and geometry are intertwined, and different sizes of the same model bike can have different geometry to produce similar "feel" for the intended rider. If you were on a mis-sized bike, in addition to having poor fit, you'd also be getting geometry designed for a person of a different size.
If you were buying a new bike, I'd suggest that a mountain bike probably isn't the best bike for you, since you don't sound like you'll spend much time riding in the dirt, but like I said, if you're happy, don't sweat it.