I figured out what was wrong and managed to fix it. It was a minor enough issue that I'm not going to return the pump.

As shown in the image above, the pressure gauge is protected by a plastic cover. It's pretty easy to slide this cover up the shaft of the pump and out of the way.
Once this is out of the way, the pressure gauge can be removed by unscrewing it counter-clockwise.
The problem is that the needle on the gauge can become loose and point to the wrong value even when there is no pressure. Firstly check if the needle is loose, re-adjust to zero and re-secure the needle by applying some downward pressure. If this doesn't work, a little pull on the needle should release it from its pin, rotate back to zero and secure the needle back into place.
Once it has been re-zeroed, you can screw the gauge back on and slide the protective cover back. Then you're done.