What needs to change with my clothing? I just wear regular street clothes now.
I find that, cycling in winter, I can just wear regular (winter or autumn) clothes: except that I have to take my winter coat off, because it would be too hot.
Also, I need warm gloves (because, unlike the rest of me when I'm cycling, my hands are gripping the cold bike in the breeze for the whole ride and doing nothing).
My toes get cold too: e.g. because my cycling shoes are extremely ventilated (my socks get soaked when it rains, for that reason). To keep my toes warm, maybe two pairs of socks instead of one.
I wear long pants when it's sub-freezing, or shorts (cotton street shorts, e.g. cargo shorts) when it's above about +5C.
I have a variety of long- and short-sleeved shirts and jerseys (some with a zip), and a wind-proof Gore-Tex anorak, for when it's raining (above freezing). I find I don't want a wind-proof (I wear a shirt and jersey instead), unless it's raining or sub-zero.
Having a pannier is useful for spare clothes: e.g. a rain coat if it might rain, ski gloves if it might go sub-zero, an extra shirt, etc. This is especially useful for commuting: where you have to be equipped for two rides at least eight hours apart, during which the weather and the time of day have changed.
low of 26F, high of 38F
So: around freezing.
Coming home on Friday (no snow or anything) it was around 0C / 32F, with an intermittent 20mph / 30kph wind. I was wearing cycling shoes, cycling-specific full-length jeans, a thin sport shirt, a zip-up thin woolen jersey with the zip partially down and the sleeves pushed up, and gloves. At the end of my hour-long commute, a group of pedestrians (all wearing coats) asked if I wasn't cold: but I was hot.
When I got home I was still warm (slightly sweating). But although the inside of me was warm and pumped, the outside of my skin was cold (even under my shirt).
Cycling can let you burn quite a lot of calories.
Also my feet were numbing by the end, and thank goodness for warm-enough (but not too warm) gloves. I'm trying some Sugoi Firewall GT cycling gloves, which have a built-in lining, and don't get too hot. I think I'll use warmer gloves, and a wind-proof shell, and do something about my feet, when it gets colder (below freezing).
Your commute is 4.5 miles, so 7 km. I cycle that in 20 or 25 minutes (to give you an estimate of how hot I'm running).
If the road surface gets treacherous (e.g. snowy) then you want to be dressed more warmly, because you dare not (for safety) cycle so fast, and therefore can't generate as much body heat (mitigated slightly by there being less wind at lower speeds).