Depending on the kind of brakes you have, and the terrain, you can use some different hand positions when using your brakes off road. The reason is that some kinds of brakes offer much better mechanical advantage, and you can brake very hard without much effort.
With caliper or v-brakes, you don't get much help, you'll probably want to brake with at least 2-4 fingers, which means you're only really holding onto your bars with your thumb and your palm, and you'll fatigue quite fast, especially if the ground is quite bumpy. Hanging on for dear life :)
With a mechanical disc brake, you can brake harder with less effort. You can probably keep 2 fingers on the bars and brake with your index and middle fingers. This is less strenuous but you'll still fatigue pretty quickly as you don't have a really good grip on your bars so you'll still be squeezing your hand tight to hold on.
The best brake is a hydraulic disc brake. When you brake with these, even on very, very bumpy ground you can use just your index finger and still brake extremely hard, maintaining a good grip on your bars with your thumb and 3 fingers.
If you're riding downhill on a big bike, and it's proper steep and rough, you're gonna fatigue quickly even with hydraulic disc brakes but that's the nature of the terrain and the kind of speeds you can reach.
As with any exercise, you can build grip strength and stamina in your hand muscles if you use them often.