I found a really cheap 12-speed SRAM groupset, and I want to know if I can just get a new rear wheel or if I need a new frame.
Your frame most likely has rear hub width of 135 mm for 10 mm QR-axles. 12-speed groupsets are likely to only be compatible with through-axle hubs of widths 148 mm (boost), and 142 mm (non-boost) and diameter 12 mm.
However, some of 142 mm hubs can be converted to accept 135 mm QR-axles by means of end caps swap. And cheaper 12-speed groupsets such as Sram NX Eagle are made to work with older Shimano freehub splined bodies that are used by 8- to 11-speed cassettes. So there seems to be a possibility to achieve what you want.
See also this discussion on the topic: https://forums.mtbr.com/wheels-tires/12-speed-9mm-qr-hubs-do-they-exist-1113587.html where some people report success of such a conversion to 12-speed. Good luck!
There's nothing about 12-speed SRAM mountain groups that would preclude them from working on a typical 8-speed era MTB frame.
Because your bike is non-Boost, you need to make sure to get the non-boost version or configuration of whatever crank you're getting.
One potential stumbling block depending on the bike is that the 49mm chainline of most non-Boost 1x cranks is already a little "out" in front. This is done in part to prevent issues with the chain wanting to catch and ride up in the small cogs. But, on a short-chainstayed 26" hardtail, if that's what you have, this can backfire and make things work badly in the large cog(s). Depending on the crank, chainring spacers and longer bolts or other hacks may or may not be able to remedy this.