I want to install a shorter stem onto my mountain bike, can I just swap them out or will I need to get the star nut replaced or reset?
As long as you're putting on the same size stem (steer tube size, not length), which you should be doing unless it came with spacers/shims, then yeah no problem.
If the clamping area is taller, you may need to get a longer bolt to reach the star nut, but then you'll also have some stem above the steer tube and that's no good either. Odds are though that you'll be just fine switching over.
By shorter stem do you mean reach (distance from steerer to bar) or clamp height? (the length of the clamp on the stem). I assume you are meaning reach, but if you are talking about clamp height then you would have to make up any difference in spacers but the answer below still stands.
You should not have to move your star nut or use a larger bolt, if you need to you are likely to end up with something unsafe. There should only be a small gap of no more than about 3mm between the top of your stem and the top of your steerer. If you have spacers between your stem and headset you can remove or swap these for a smaller size to make the stem fit.
The star-nut/ top cup and bolt acts only to provide preload-tension so that you can clamp the stem onto the steerer without movement in the headset. Once the stem is on you could technically remove the top cap (but no one does this). If you are having to change these then it is likely that the stem won't have adequate space to clamp onto the steerer safely. Spacers are there to provide leverage for the top cap to add tension but the stem can sit directly on top of the headset if required as long as you have that small gap.
Other things to be wary of when changing stem:
(sorry I can't add images from this PC due to security restrictions)