The normal process would be something like:
- Fit the new rear derailleur to the bike without crossthreading the bolt. Then thread the chain through the jockey wheels, and join it. (if you've changed the toothcount then you may need a longer/shorter chain, but that's a tangent)
- Set the high and low limit screws on the rear derailleur so that the chain can't go too far. Do this by pedalling the crank by hand slowly, and push the derailleur with your other hand. The chain should move up and down but not over the end of the cassette. If it goes too far, or not far enough, then tweak the limit screw by a half turn. Experiment to find the right rotation.
- Fit your inner cable - start by setting your chain on the second smallest cog in the cassette. Then move the shifter into the highest gear (the hardest one) without moving the chain or wheel. Set all your barrel adjusters to be closed, and then back them out a turn or two.
- The inner cable should come through the outers and then through a pinch bolt on the derailleur. Pull the inner cable taut and pinch it down with the bolt.
- Test turn the crank by hand - your chain should move to the smallest cog as the minor slack in the inner cable is taken up.
- Try shifting while slowly hand-pedalling, up and down the entire cassette. If the chain seems to be struggling to get up the cassette, then back out a barrel adjuster a half-turn, and vise versa.
NOTE: This is a finger-chomping area. Between the chain and anything it runs on, as well as the wheel spokes and the frame. So pedal slowly, use the rear brake to stop the wheel not your hand, and be mindful.
You might not get it perfect, but as long as shifting happens then you can move onto a riding test. The bike will be subtly different when under tension, so making it perfect in the stand can be a waste of time.
Also, after a month or so the gears will un-adjust themselves as your new inner cable settles down. The metal wire strands compact slightly, making the wire slightly longer which throws your adjustment off. So just twiddle the barrel adjuster out some more to restore shifting.