I crashed yesterday and damaged this aluminum frame. At first I thought I shouldn't worry, but giving it a second thought, I think I should just replace before it breaks while riding. I would say it's like one mm deep.
In the long term I think you've written off your frame. Sorry.
We've figured out that this is the underside of a chainstay.
Chainstays live mostly in tension while being ridden - that means you're pushing down on the middle of the bike, and the two wheels "want" to roll apart, away from each other.
What you've got there is a weak point, which will act as a stress riser and the chainstay will eventually fail at that point.
The good side is you have two chainstays, so sudden failure of the damaged one will be bad but probably won't collapse your bike all at once.
I would continue to ride the bike, but keep a close eye on the area weekly and once it changes, then immediately stop riding. During this time, look for a replacement frame or bike. Take photos on your phone periodically to compare - the mind forgets as time goes on.
If you are attached to the frame, it may be worth getting a framebuilder's opinion. Being aluminium its harder to fix than steel or carbon fibre due to heat treatment requirements and not damaging the rest of the frame. But a chat and a quote is free, and gives you a possible dollar amount that is the "sunk cost" and the difference between that and the new cost of a new bike is the "opportunity cost" for justifying a cheaper upgrade.
Show a bike mechanic to be sure but the frame is likely to be OK unless you like to jump a lot.
Judging from plastic pedals, 9 speed or less chain, and V brakes, you probably have a relatively cheap frame which means thick straight gauge metal.
You could gently file or sand the surface of the scrape flush with the rest of the tube. I’d leave it unpainted or coat with clear nail polish so you can periodically check for any cracking.
Alloy frames can take quite a cosmetic beating, it’s dents you need to be more careful with.
If this is actually a higher end frame then the tubing will be thinner and the frame is probably dead and needs to be replaced.
This may not add that much compared to the existing answers. However, here are some thoughts.
We don't really know how deep the scratches are. Just surface scratches are probably not worrisome. Also, we don't know how thick the chainstay is where it was scratched. If this were a racing bike with very thin tubes, I'd lean towards replacing it as soon as practical. If that were carbon in general, I'd take it in for a repair immediately. Thicker tubes would give you more safety margin, but again, we don't know how thick they are. Generally, a lower end bike will probably have thicker tubes.
We don't know exactly how much stress that area of the bike is under. I stated in comments above that this looks like the center of the chain stay on the drive side. Generally, the drive side is under more stress because that's where the drivetrain is, so the force from turning the cranks is being transmitted to the frame. Generally, I think that areas closer to the ends of the tubes are under more stress. Areas under more stress from riding will probably fail faster.
The concept of stress risers or concentrators is applicable here. Basically, the gouges probably act as stress risers, like Criggie's answer says. This is why everyone is saying that the frame could fail down the road.
Overall, I'd second Criggie's recommendation for watchful waiting. You do want to be looking for cracks propagating from the damaged area. If the chainstay fails, I agree that it shouldn't cause an immediate loss of control; in contrast, failures of a fork steerer, fork blade, or downtube would cause a loss of control. If the scrape is really 1mm deep at the worst point, then you're probably justified in trying to see if you can get a replacement bike or frame.