This does not sound like a basic derailleur indexing issue, nor a bent derailleur hanger problem.
Possibilities that I can think of:
You have a broken or malfunctioning shifter. As you try to change to the larger sprocket the shifter is pulling the cable, the ratchet mechanism is not catching and is slipping back two places.
Your derailleur low limit is set too far in. As the shifter pulls cable it hits the limit, does not complete the shift and again slips back two places.
B-screw is badly adjusted to the top jockey wheel hits the 32 tooth cog and has the same effect as the low limit set to far in.
You are hitting the upshift lever while working the downshift lever somehow.
I suppose it might also possible that your indexing is off by one gear to the high side, so when you try to shift to the lowest ratio the shifter is already in it's lowest position, resulting in an upshift instead.
What I'd do to troubleshoot this is disconnect the cable from the derailleur, With the bike is a stand make sure the derailleur can shift onto all sprockets by pushing it inwards while manually pedaling (be real careful not to catch you fingers in the chain). Adjust limits or b-screw if you find and issue.
Grab the cable with some pliers, apply tension, work the shifter. Can you make the shifter slip on the last downshift position? If this is the case it's very likely the shifter is malfunctioning.
If you don't find a problem try reconnecting the cable and working through the Park Tool rear derailleur adjustment procedure to ensure correct limits and indexing.