As mentioned here earlier, an alloy handlebar works like a heatsink. Even with nicely padded pogies and with lobster gloves, silicone grips do a lousy job of insulating the heat. They slowly transfer the heat from hands to the center, where the stem and fork act as heatsinks.
One solution is to switch to a carbon handlebar, but perhaps a milder solution using grips with better insulation will do.
Silicone mittens are normally sufficient to remove a pot from a 400F/200C oven, but hold the pot for a few seconds, and the heat will soon reach your fingers.
Still, silicone may have been just "good enough" because the temperature difference between a pot out of the oven and my hands far exceed the temperature difference between the ambient riding temperature (-2C to +2C) and my hands, but I can now say from practical experience that simply holding the grips for two hours does drain the heat and one's fingers will start to freeze.
What is a good grip material for heat insulation?
Should you just "toughen up"?
How critical is it to get insulation right? If you're embarking on frostcycling, before listening to someone derisively and dismissively writing about toughening up, consider for a second that they may not actually know what they're talking about, and read carefully about frostbite.