My 26" mountain bike currently has a 2x10 gear setup with SRAM components: a SRAM S-1250 crank with 38/24 chainrings, and a SRAM PG-1030, 10-speed, 11-36t cassette. I feel that when riding on non-technical dirt trails I am maxing out my drive-train, meaning that I am not able to accelerate beyond a certain speed when riding the 'hardest' gear. I have been reading about the Shimano 3x10 system which may allow me to reach higher speeds, due to higher possible transfer ratios. So, how much faster can I with a 3x10 setup? please elaborate on the math in your answer...
Well, if you switch to 3 in front, and then make it a 52/38/24 set (equal steps) then your theoretical new maximum speed would be:
[new max]=52/38*[Your current max]
Currently at your top speed one revolution of your crank leads to 38/11th of a revolution of your rear wheel. This will become 52/11th or about 1.37 times as much.
This is not the whole story however. You probably know that wind resistant is much more of an issue at higher speeds than rolling resistance. So (unless you are really strong) your actual maximum will be quite a bit lower than the theoretical maximum calculated above. This tool allows you to set your speed or power output and calculate the other. If I assume your current max as 30kph, your bike weight at 9kg, your weight at 70kg (don't be offended, I'm just guessing), and you are on MTB tires and in the "hoods" position (whatever that is) then your power output at that speed would be about 220 Watt. With the extra chainring in front your theoretical maximum speed would be 41.1kph (all other variables being equal) but to attain it you would need to about 470 Watt! More than twice as much!
Obviously in your current situation you are not pedaling efficiently at your maximum speed because you are spinning your legs like crazy, but a 1.4 times speed gain is probably too much to hope for. As a test to see where you can get you could ask your friendly LBS owner if he'll lend you a bigger front chainring for a test ride and see if you can find out from that at what point you max out.
Assume your wheels have a circumference of l = 2 m. With a 38x11 gear you will cover
d = 38 / 11 * l = 6.91 m
per crank revolution. Assume that you can pedal at a cadence of 90 rpm, that will give you a speed of
v = d * 90 rpm * 60 min = 37.3 km/h
On a road which allows for such speed you should be able to pedal at a higher cadence if you have good riding technique, i.e. achieve even higher speeds.
The maths don't care about brand names, but take a Shimano XT triple with 42-32-24 teeth instead:
d = 42 / 11 * l = 7.64 m
This results in
v = 41.2 km/h
at 90 rpm, which is
(41.2 * 100 / 37.3) - 100 = 10.6%
faster. Actually, the ratio of the two different big rings will give you the same
(42 * 100 / 38) - 100 = 10.6% as well.
If you keep your 11-36 cassette, the drive train will have a total capacity of
(36 - 11) + (42 - 24) = 43 teeth, which may be more than what your rear derailleur can take care of.
Apart from that it is often easier to maximize downhill speed by optimizing aerodynamics instead of pedaling.
I am going to skip the gear ratio discussion because others have covered it, and it really isn't the most important consideration.
If you're spinning out frequently, you'll want a higher top gear, but you can only go so high with those cranks-- probably 42, max, on the big chainring. That might be enough to keep your from spinning out while still retaining the capabilities of the bike.
If you decide to swap out the cranks/chainrings and get something more like a road-bike or cross bike, you will mess with the intended design of the "mountain" bike. There is nothing wrong with doing that but the gears are low on a mountain bike for very good reasons like negotiating stumps, ruts, rocks, mud/gravel and even very short 100% grades.