The screw that you are missing there is the B Tension screw. This screw rests on your derailleur hanger on your frame and sets the angle of the derailleur body.
Usually, when you put a new derailleur on a bike you use this screw to adjust the distance of the upper pulley (Jockey pulley) on your derailleur up or down from the teeth on your cassette. You want the top pulley to be 5-10mm away from your cassette teeth when then chain is on the largest / easiest gear. This keeps shifting nice and tight, because the derailleur is as close to the cassette as possible, without riding the pulleys on the cassette teeth.
Screwing this in will move the derailleur body and pulleys downwards, away from the cassette, while screwing it out (or completely out in your case) will cause the derailleur body to be pulled up by it's spring into the contorted position you mention.
Read more about it here (scroll halfway down the page)
Also, if it wasn't obvious already— you've now learned why you shouldn't spin the rear wheel while you adjust the High Limit screw...