I’ve fallen a couple times, but no serious damage is visible on the helmet except for a couple divots here and there.
Should I consistently be replacing my helmet after a certain amount of time? After every relatively serious fall?
Helmets should be replaced roughly every 5 years and after any crash where your head makes contact with the ground.
Helmets will crumble, compress or otherwise deform in sometimes hard to see ways when you hit the deck. The structural integrity of the thing will be massively diminished after even a relatively tame fall. This is one area where you don't want to mess around much.
When to replace a bicycle helmet:
Why are you using a helmet?
If you're using it because you're doing something which involves a decent probability of you falling and hitting your head, then you should probably replace it after every serious incident. Some companies (e.g. Giro) will give you a discount if you trade in a damaged lid, so that they can learn from its demise.
If you're commuting or doing something less dangerous, but which offers some scenarios where you think a lid will assist, then you might be less concerned about replacing after every ding. But the helmet is designed to work when pristene; as soon as there is damage, any damage, it won't act in the way that is was designed to, so is going to offer you a sub-optimal service.
But if you're just using a helmet because you think you should, although remain generally unconvinced (peer pressure, race rules, certain state/national legal obligations, enhanced air flow around the scalp), then don't worry about it ...