I have managed to put a small inwards dent in my aluminium mountain bike rim. I know that there are specialty tools for fixing dented rims, such as the Morningstar Rim 'Rench and the Wolftooth 8-bit rim dent remover, but is it a good idea to use such tools to bend the rim back? Or will it weaken the rim so much that I should just leave the dent be?
There's not an answer that applies to every aluminum rim because they vary a lot in the ductility of the material. The high quality lightweight modern rims are very hard and strong but don't have much tolerance for getting bent around without creasing or cracking. Other aluminum rims toward the more play-doh end of the spectrum (generally less sophisticated alloys and heat treatment) are more malleable and can sometimes have dings bent out of them and have it be almost imperceptible. And of course the former is much more likely to be critically weakened afterward then the latter.
Also, the sidewall of a tubeless rim is much shorter and stiffer than non-tubeless. The force required to get them moving is higer. In my experimenting with trying to take dings out of various discarded modern tubeless MTB rims, I've found it's pretty easy to wind up with a rim that has a creased and micro-cracking area that I would generally trust much less than having left it alone.
For most minor dings on a mountain rim that aren't noticeable when you ride and you can see aren't hurting the tire seating, you're better off leaving them alone since there's simply not much to gain by trying. And for more major ones, you should be very prepared for failure now or later, and so there isn't much use trying in that case either. This is the reason why bending out rim damage isn't a very commonly done repair, other than to get home on a ride when a tire won't seat.
In my experience, where taking out dings can have a place is on very basic rims and especially in low-resource environments. A minor ding on a rim brake rim is still a significant problem, and in some cases it can be made pretty good again by bending it out on the backside and doing some sandpaper/rim eraser work on the top side. When the structural damage was minor to begin with, the increased odds of failure seem pretty low in practice after doing that, although they do still exist and so there's an element of risk involved, and the person riding the bike should be accepting of that.
I have saved and ruined rims by attempting repairs.
The main risk is that Aluminium work hardens quickly, so one bent cannot generally be bent back to original shape without weakening and distorting the metal.
I have torn a rim by bending it back, and I have snapped a slice clean out of a cast aluminium rim. I've also recovered a rim and had another 6-12 months usage out of it before finding a cheap replacement wheel.
If you have rim brakes, the brake track needs to be flat. Any lumps and dents will exhibit as a pulse in the brakes when activated. It will also accelerate wear of your brake pads over time, and the change in friction every rotation will make the braking effect lower.
If you have disk brakes, their effectiveness will not be altered by rim damage.
Any scratches or burrs that are in contact with the tyre should be filed out then sanded smooth before they eat holes into the rubber.
I would anticipate that a failed repair could require replacement, and price up a new rim or wheel. Then have a go at minimising the dent and if it fails, you needed a new rim anyway.
You should definitely at least attempt to repair it, although if it's too bad then replacing the entire rim is the better option.
If the dent means sidewall is bent, a large adjustable wrench can be used to carefully bend it back straight. You may need to do the bending on several places, like you would bend a brake rotor with a rotor straightening tool.
If the dent is elsewhere, a hammer and a block of wood can be used to hammer it back. You must first relax the spoke tension by screwing all nipples back two turns. Then you can repair the dent. If the dent is a radial dent at the spoke wall, you must entirely remove the spokes at the area so that you can hammer it back. In this case, you need three blocks of wood: two below the rim (on the tire side) on either side of the radial dent (so that the rim is a "bridge" over the two supporting wood blocks), and third above the rim (on the spoke side) to protect it from the hammer. Then put back all spokes and tighten the nipples again and retrue the wheel.
If you don't have wheelbuilding skills, the book to buy is The Bicycle Wheel by Jobst Brandt. It describes how to build and true reliable wheels, and also the dent repair is explained there.