A bent hanger usually manifests as issue mid cassette. If that's the issue (i.e. limits are set correctly and first and last gear work fine) then it's a safe bet the hanger is at issue. If you can't tell by looking then the only real way to verify it is either replacing the hanger with a known straight hanger or using a derailleur hanger alignment gauge (e.g. Park Tools DAG).
If you have to re-adjust your limits due to the chain going over the cassette, then your derailleur is most definitely bent.
As a MTBer running a 1x12 drivetrain a derailleur hanger alignment gauge was a necessary investment for me to keep my drivetrain running smoothly.
One other possible issue is cable housing. If the cable was replaced but the housing was reused, then there can be friction that causes issues. My experience with that has been mainly delay downshifts (the derailleur is slow to release).
If you are using a Shimano clutched derailleur, make sure you turned the clutch back on. Forgetting to turn it back on can cause shifting issues which the chain bounces around on trail.
Update (after more info from Ender):
If you are unable to shift to the largest gear on the cassette and your barrel adjuster doesn't have enough range, then it's a cable tension issue. It could be the new cable stretched. In any case you can fix it as follows:
- Shift to the smalls gear on the cassette (i.e. relieve all the tension from the shift cable by shifting down all the way).
- Turn your barrel connector all the was in, then back off a little (to give yourself some adjustment range to loosen slightly if need be)
- Unscrew the bolt holding the shift cable to the derailleur and loosen it so it is putting no tension on the derailleur (it won't be in your case since it's too loose, but I've had a bike come from the factory too tight so as a general approach it's a good idea)
- Verify your limit screw is still set ok (visually the derailleur should still line up with the small cog and you should be able to pedal the drivetrain without the chain dropping or shifting up)
- pull the shift cable taut and tighten down the bolt on the derailleur. When you pull, pull so it's taut and not causing tension, you don't want to to be pulling the derailleur up to the next gear. If you left a bit of back adjustment in the barrel connector, it's not that big a deal if screwing the cable down makes it a bit too taut (you can just loosen at the barrel)
- Carefully shift to the largest cog (incase it goes over), tightening the barrel connector if need be to get it to go all the way up. Then adjust that limit screw.
- Shift through the entire range to make sure every gear is working properly.
Don't worry about messing anything up. If you haven't done derailleur adjustments it can be a bit intimidating but anything you do can be undone. A 9-speed drivetrain should be fairly easy to get dialed in. 12-speed can be an act of frustration as the tolerances are much tighter.