I am going to assume we are dealing w/ a JIS style square taper bottom bracket for this issue. At least, I am going to address it as such.
If it is a cartridge style BB you really have no choice in repairing the less than perfect feel so either accept the roughness and look at a new one next season or switch her out when you feel.
If it is a loose ball and you are partial to it (ex: ti spindle) the roughness is a sign that you should get on servicing it asap. The roughness can come from contamination of the grease and result in the spindle race pitting and the cup races pitting as well. Once these are pitted you will most likely never get it back to that silky smooth feeling you so desire.
In reality, chances are it is a cartridge JIS BB.
Domsterr is spot on with the 'lateral' play issue. To add to the recommendation of spinning the cranks I would add the following:
take the chain off of the chain rings, when giving it a spin, as the chain will interfere with your ability to assess by ear.
hold the down tube or seat tube and 'feel' for the grinding through the frame. You would be surprised how effective that is. I use it regularly for assessing hub condition.
That's about all I've got.