I figured out what was wrong and managed to fix it. It was a minor enough issue that I'm not going to return the pump.

As shown in the image above, the pressure gauge is protected by a plastic cover. It's pretty easy to slide this cover up the shaft of the pump and out of the way.
Once this is out of the way, the pressure gauge can be removed by unscrewing it counter-clockwise.
The problem is that the pin indicating the current pressure can slide up its axis to a point where it is going to be turned by the axis when the pressure changes, but it won't start at 0. The way I fixed it was by jiggling the guage until the indicator fell back into place where it should be. I suspect it came loose when I biked the pump across town.
Once it has been re-zeroed, you can screw the gauge back on and slide the protective cover back. Then you're done.