I had to adjust a handle bar of an older road bike I want to start using again. When unscrewing the handle bar to stem bolts I noticed that the washers had grinded into the aluminium already. As I had to readjust the handle bar a few times this got only worse to the point that the washers were already deforming and locking on the screws. I fixed that gently with a pipe wrench and finally reassembled everything back together with proper fit and almost equal gaps. Is the worn material or now slightly different fit of the screws of any safety issue or could it lead to any or is it just a cosmetic thing? By the way I only used a really light torque of 4 Nm.
Damage to the faceplate from the washers digging into or not being present at all when tightening is pretty common and not an issue unless taken to an extreme.
If the washers are mangled and not functional, it can be good to replace them. It might also work to re-orient them and/or put some grease between the bolt shoulder and the washer.
Don't undertighten the stem bolts. That's even more true if the washers aren't doing their thing; when they're staying put properly and act as the bearing surface for the bolt shoulder, they increase the clamping force for a given torque input, which means that for the same given torque input there will be typically be less clamping force if they're rotating under force and digging into the faceplate. If you can't stop them from doing that then use normal torque and grease both sides of the washer.