I've been searching on Google for the last hour trying to figure out if this is some kind of security nut. It's on a Shimano hub on the rear wheel.
Its a rear wheel security axle nut, with an anti-rotation washer underneath. I see a band brake or a roller brake, and a Shimano Nexus (8?) internally geared hub, which are reasonably valuable, hence the security nut.
If you want to get it off with a minimum of damage, I'd make a tool.
- Get an old set of bootpinch pliers and round off the cutting faces with a grinder so they match the profile of two opposite indents. Then unwind it like a wrench.
- You could make some sort of a socket to go over the nut - but this would be difficult, and I'm not sure how one might make the lobes.
For a once-off I'd use a Pipe Wrench or some Sliding Jaw Pliers and marr up the surface, with the intention of replacing it with a normal nut.
Also, if you have some round wire that is the same diameter as the indents, cut four short lengths and tape them into the indents. This will take the load/pressure of your tool off the outside surface of the nut.
Some people might attempt to wrap the nut in cloth or leather or something to protect the surface from the tool, but personally I find that rarely works for me.
You could possibly grind some flats into the nut, but that will weaken it, you risk damaging the frame, and won't be a great long term solution compared to just replacing the nut.