The frame has been cold-deformed because someone's used too much torque to tighten wheel nuts which has squished the metal over time. Either the dropouts are too soft/thin or someone's used an impact driver or a cheater bar on the bolts.
You're not going to get a thick anything on the outside of the axle's bushings AND inside the dropouts - this is the Over Locknut Dimension (OLD) and will be fixed at 130mm or 135mm or maybe 126mm. If you shove the equivalent of a 2mm plate on the inside with a washer, then the rear dropouts will be forced apart, and will no longer be parallel. This makes your rear wheel get loose even faster. Not recommended. Also probable not needed because the axle's bushings are the right size to fill that curved-out area of the dropout.
A washer on the outside of the dropout is a reasonable idea and should work well. You want something that presents a flat surface under the axle nut, but is profiled to match the frame. Hopefully your axle has another 2-4mm of spare thread exposed.
V _ +---+
_|| | /
| | | /
|OO| |O| <-- axle hole
|_ | | \
|| | \
+---+ This side touches bike
To stop your rear wheel moving in the dropouts, you might want to fit a Chain Tug on the chain side, or both sides. These are mostly built for bikes with trackends, but some do exist for dropouts like yours. One example is the "Surley Hurdy Gurdy"
More related info Are there tugnuts made to fit forward facing horizontal dropouts?
You could put also the axle further forward or backward in the dropout. This lets it sit against "clean metal"
Backward/up the dropout helps with climbing angles and makes the bike longer, better for long+straight rides and improves function of front brake. Also it will support the nut on about 3/4 of the circumference. Also you get a positive stop which helps a lot every time you reinstall the wheel.
Also improves "braking" if this bike is a fixed gear and you elect to skid-stop.
Moving axle forward in dropouts means a shorter bike wheelbase, leading to snappier turning, and maybe fewer links in chain, but the wheel is poorly supported by the longer finger of the lower dropout arm. Wheel reinstalls become more fiddly to get everything aligned right.
Ultimately, these stamped-metal dropouts are a bit weak. Your more-expensive solution is to talk to a frame builder and get harder/thicker ones installed.
If you're competent with a brazing torch, you might be able to do this yourself, though alignment must be perfect and not move throughout brazing.
Here's a higher quality dropout example - notice how both sides bulge outward? This is a thicker insert that has been added and brazed into place. I bet the inserts are somewhat hardened too, whereas yours are plain mild steel.
It may be more viable to simply replace the frame with a used one of better quality. Keep your eyes open for options, and you might be able to move all your parts from the current frame to a replacement.