The frame itself is no concern at all. You are likely remembering statements that aluminum has a fatigue life but steel and titanium don't. What this means is that the latter materials will never fail from fatigue if the loads on those materials are kept below a defined limit. It's theoretically true that aluminum will eventually fatigue. However, a well-made aluminum frame isn't likely to fail in a rider's lifetime from fatigue in the first place. It's true that if you are traversing rocky gravel, the loads on the frame may be greater than originally anticipated, but again, I doubt this is a practical problem.
Additionally, if you are thinking of operating the bike on smooth dirt roads, I'd expect the loads on the frame to be pretty much what you would experience on tarmac - in fact, good dirt roads are probably smoother than rough tarmac. And that raises another issue: "gravel" has many definitions. You can ride a lot of road bikes, even older ones with 25mm tire clearance, on smooth dirt. However, even current generation road bikes tend not to clear more than 32mm tires (there are exceptions). That is, there won't be space to put 40mm tires in the frame. That limits the off-road terrain you can go on with a road bike.
Here is one attempt, by Cyclingtips, to grade gravel surfaces by roughness, and to suggest a tire size for various surface types. It may have been originally written a few years ago. Many riders in 2021 or 2022 would probably fit wider tires than they specified on their gravel bikes. Conversely, you can usually get away with narrower tires than they stated. One exception would be for borderline MTB terrain. I'm not sure their photo example really does that justice, but if you are on very rocky terrain, I think most people would use tires well over 40mm if their bike can fit them.
Last, if you are really sticking to smooth dirt, you could just fit a wider slick tire without tread. Give it a try, you may be surprised. This may not work so well if the dirt is wet, and not everyone will necessarily be comfortable with this (and that is OK). I do have an older road bike with 25mm tires, and it's been ridden on smooth dirt or limestone paths fairly often, so I can attest that it's possible.