Carbon fiber isn't necessarily a "weak" or "fragile" material. If you had a tube of the same diameter and thickness of typical CF as a typical steel frame tube, that CF tube would be extremely strong and durable.
Metals like steel and aluminum are isotropic materials. That means their mechanical properties are identical in all directions. If you have a cube of steel, it will respond the same way regardless which direction you pull or push on it.
Carbon fiber is a composite material. It consist of tons of little bundles of the fibers held together with an epoxy.
A block of steel is, well, like steel, but carbon fiber is like a big bundle of straws glued together. In one direction, it is extremely strong, but if you push or pull to the side, it will collapse. In that one dimension where it is strong, it is vastly stronger than steel. However, in other directions it's rather flimsy.
So, engineers have been able to exploit those properties in bicycle frames. In a bicycle frame, the vast, vast majority of the forces are primarly along a single dimension. They can make tubes thinner and lighter yet still retain the desired strength and stiffness.
So, there is no mechanical reason that you couldn't build a fully loaded touring bike or something like a Salsa Fargo with a carbon frame, and it could be just as tough and durable. And it would probably be lighter than a steel or aluminum frame. But the reason it isn't done is because of the market. Carbon fiber is an expensive material and difficult to work with, and its mechanical properties are best suited for when you demand very light applications.
When you build a steel framed bike, when you get the tubes sufficiently strong enough along their length, that because of steels isotropic properties, you get the lateral strength for free, the strength to resist things banging into it, withstanding crashes, etc.
In a carbon fiber frame, you don't get the strength in the other dimensions unless you choose to design it in. In carbon fiber bikes, where weight is a serious concern, the engineering decision has been made to not make the frames strong in those areas. They could do so, but they choose not to because its not necessary for the bikes intended purpose.
When you build a heavy loaded bike, you lose a lot of the carbon fibers advantages, and so it would be far more economical to use steel or aluminum. Especially when throwing a couple filled water bottles in your pannier nearly exceeds the weight savings.