Space for squirm, plus tolerances.
Firstly, cranks are made in many different shapes and styles, and not all are the same thickness. So some slack is left to allow for this. Imagine how annoyed you'd be if changing cranks meant a new BB cartridge or axle?
Secondly, modern funny-shape crank/axle joints are less prone to this, but a square taper fitment would creep further onto the axle over time. The more times the crank was removed, the further the crank can move up the square taper. We're only talking fractions of a millimetre.
So if your previous crank was original on the bike with the bottom bracket, they may have been better matched and the gap could have been smaller.
No you don't need a spacer - the axle does all the load bearing by itself. A spacer would simply add to friction and weight and provide no benefits.