I know that the two main categories are the mineral oil (Shimano, Magura) and the DOT type (Avid, Hayes).
But what is the difference between the DOT 3, DOT4, DOT 5.1? Which one should I choose for average mountain biking?
Part of the difference is boiling points:
Dry boiling point Wet boiling point DOT 3 205 °C (401 °F) 140 °C (284 °F) DOT 4 230 °C (446 °F) 155 °C (311 °F) DOT 5 260 °C (500 °F) 180 °C (356 °F) DOT 5.1 270 °C (518 °F) 190 °C (374 °F)
3, 4, and 5.1 are glycol-based, and must be changed periodically, as it absorbs water. 5 is silicone-based and does not need to be replaced.
To answer the "Which one should I choose for average mountain biking?" part,
You don't really "choose" your brake oil, it's specified by the manufacturer. The oil is chosen depending on the heat output of the brake and the general construction of the system. A different oil than what the manufacturer recommends might affect/break the cables, seals or some other parts of the system. Check your brake's manual (most likely available online if you lost it) to know what you need.