It sounds like your headset is not adjusted. Your stem cap is used to adjust this. Specifically, the more you tighten the stem cap, the tighter the bearings will get, and the less play you will feel.
Here is a picture from Sheldon Brown's article on adjust headsets:

My process for checking headset adjustment:
- Check the headset adjust:
- Grab the front brake with your left hand, and grab the interface between the frame and the top of the headset with the other. This is the "No Wrench Flats" in the image above.
- Rock the bike back and forth with the brake on.
- If you can feel the top of the headset moving independently from the head tube, it probably needs to be tightened.
- Pick the front end of the bike up by the top tube, and let the handle bars/front wheel/fork "flop" from side to side. If they flop around loosely and don't bind up, you haven't over-tightened the bearings.
- If the headset is out of adjust:
- Crack loose the stem Pinch Bolts. Don't take the stem off, just get it loose enough to slide around. When locked down, the stem will hold everything together, so we can't adjust anything.
- Tighten the stem cap bolt (labeled as "Top Cap" in the images above) by about a quarter turn each adjustment and then check the headset again.
- If you have over-tightened, back it off by an eighth turn and check again.
- Once everything is adjusted, line up the stem with the front wheel and lock back down the Pinch Bolts.
- Give it one more quick check (nothing should have changed, but it takes two seconds, so why not?).
- Best part: Test ride! See how she feels.
Once you've gone through the process a few times, it gets to be very quick. I usually check my headset ever week when I do a quick once-over of the bike.