I an not talking about upgrading. How do you know when to buy new wheels because the other ones have worn out? How do you tell a wheel is worn out? I can always true a wheel back to true. And nothing is dented, but the maintenance is getting a little more than it should be.
At some point the spokes become too fatigued and start snapping.
At some point, with rim brakes, the rim wears through until it's no longer stable.
At some point the bearing races in the hub become too worn.
Of these, only the third reasonably demands that the wheel be replaced -- wheels with worn spokes can be relaced, and rims can be replaced without relacing. But in some cases, especially if more than one of the above problems is looming, it may be more cost-effective to replace the wheel rather than repair it.
And, of course, there's always the case of some sort of actual damage to the wheel -- ripped spoke holes in the flange, bent flange, "taco-ed" wheel where both spokes and rim will need replacing, etc.
There are several reasons why you might need to replace a rim or a wheel.
Of course, you can replace a hub, spokes, or the rim individually. If you replace the rim, you should replace the spokes. Why risk building fatigued parts into your new rim? And if you replace the hub, unless the wheel is very new, you should build with a new rim and spokes, for similar reasons. If you can do the work yourself, you will find it cost effective to build a wheel by hand, but if not, buying a replacement wheel (unless it is a high end hand built wheel) is usually more cost effective.
Given your description, I think number 4 is your issue. I'd say time to replace or rebuild the wheel.
I hope that helps.
Most reasonably new rims come with a wear line. There should be a groove machined into the breaking surface. Once you can no longer see this, the brakes have worn down the rim too much and the rim needs replacing.