The basic arrangement for cross-laced spokes is thus:
Spoke Length = sqrt[ (RRSP - (HSR * cos(SAA)))^2 + HFO^2 - (HSR * cos(SAA))^2 ]
For radial (straight) spokes, the formula is simpler:
Spoke Length = sqrt[ (RRSP - HSR)^2 + HFO^2 ]
- RRSP (Rim Radius plus Spoke Penetration) is half of the Effective Rim Diameter given by the manufacturer plus 2mm for the spoke's penetration into the nipple
- HSR (Hub Spoke Radius) is the radius from the hub center to the outermost edge of the spoke holes on the hub
- SAA (Spoke Anchor Angle) is the angle of from the hub spoke hole to rim hole that the spoke goes to. It depends upon the lacing pattern and which individual spoke you are calculating for.
- HFO (Hub Flange Offset) is the distance from the hub's flange to the lateral center of the wheel, based upon the dish of the wheel. For a front wheel, this will be the same on each side, but for a rear wheel, which has a dish to accommodate the rear gears, this is different for each side since the hub's own lateral center is different from the wheel's lateral center.
I'll credit this page for the formula; it also has more detail.
There are also a bunch of calculators online, the easiest of which is probably Damon Rinard's spocalc spreadsheet. Other calculators:
As usual, Sheldon Brown has a great page on wheel building and you can learn more about the math from the above mentioned book.