Saddle setback is a fixed position for each rider.
It is necessary to place the saddle in the position which places your knee in alignment with the pedal spindle, when at the correct saddle height.
Arbitrarily changing the saddle setback can interfere with a proper fit.
You should set your saddle height, then fore/aft position to align your knee correctly, then choose a stem length that makes the saddle position reasonably comfortable.
If the 20mm setback post is necessary to align your knee, then, yes, you may need to subtract that extra distance from your stem. But that will leave you with a very short stem in this case, which will affect how the bike handles.
I would expect that if that change is necessary, you are on the wrong frame size.