I would quantify the risks differently from Nathan. Remember that the fork steerer is hollow. The compression plugs support the walls of the steerer from inside against the stem's clamping force, which is actually a requirement because the carbon isn't strong enough to be clamped unsupported. Also, the potential issue is that initial damage to carbon can be invisible, and it can propagate with time. Hence, I'm not sure we can say the chance of damage is distant.
That said, components are all designed with a safety margin. For example, people may not have perfectly calibrated torque wrenches, or may tighten the bolts in the field, or they may mix the sequence of operations up as you did, etc. Engineers know this, so we generally should not expect the steerer to crumble to dust the instant you remove the plug.1 Also, you exposed the steerer to the unsupported clamping force for a brief period.
Unfortunately, the ideal case is that you or someone would inspect the steerer with ultrasound or some other advanced imaging. You presumably don't have the equipment at hand or the skill to interpret the readings, or you wouldn't be asking the question. While this answer may be vague and not fully satisfactory, I'd just urge watchful waiting. This involves periodic inspection. You would be looking for cracks showing on the steerer, or changes in feel at the front end. You could look at the steerer every week for a few weeks, then less frequently thereafter. You could take a reference photograph now so you know a surface feature was there a few weeks ago.
Footnote 1: Ultralight items are generally going to have much thinner safety margins. I assume you don't have one of these. Anyway, it's worth mentioning that the advice would lean much closer to replacement if this were an ultralight item.