That's an indication of possible crash damage, because the brake arms have bent and distorted their mounts. However your photo shows rear mounts which are rarely damaged this way.
It can also happen if someone deforms the mount by using the wrong bolt, or over-tightening it and deforming things.
You may be able to use a small automotive bearing puller to pull the brake arm off. A couple of flathead screwdrivers can help provide leverage.
Another moderately good help is to warm the area with a hairdryer on high. This may be hot enough to damage plastics, but shouldn't damage paint. If the frame is carbon fibre or aluminium alloy then don't try heat, but it works nicely on steel.
Last resort is to cut the brake arm off using a dremel and small cutoff wheel. Its very hard to avoid damaging the mount posts though.
Crazy thought - have you tried backing off the spring tension adjust screws? Not all the way out, perhaps 10-12 mm.
If the tension springs are somehow locked in a groove or something, this might help.