Over time forks become less responsive (move slower basically), but as this effect happens over the course of 12-18 months (the usual fork oil and seal lifetime), it's difficult to notice as you just get used to it. So most fork manufacturers tell you to change it after so many miles.
This responsiveness loss happens for a couple of reasons on different types of damping systems, on mine (Boxxers) it's because of pollutants in the oil itself - 'dust' from the bushes, dirt, seal flake off, etc.
You can get a rough idea of the state of your fork oil just by looking at it, new stuff is clear tinted yellow. I've seen fork oil come out looking like mercury from the sheer quantity of bushing dust in it...
Technically you only need to change seals once they start leaking oil, but of course that usually means that they've already failed their job - if you ride a lot in dirty conditions (which you should be!) they want changing every year.
Bushes only need changing once there's excessive play between your stanchions and sliders, this usually takes years but you can test the same way you would for testing a headset.