Bike fit can affect breathing mechanics if the bike fit puts you in a position that impacts diaphragmatic breathing (i.e., "belly breathing"):
Ventilation for the endurance althlete is most effectively performed through diaphragmatic breathing, in which contraction and relaxation of the diaphragm pulls and pushes air from the lungs. Secondary muscles of ventilation include the inter-costals (between the ribs), the abdominal musculature, the trapezius, the levator scapulae and the scalenes. If diaphragmatic breathing is compromised these secondary muscles can become chronically overworked leading to myofascial type pain (i.e., pain between the muscle and the muscle covering). This is often seen in the upper neck and shoulder muscles.
-Phil Burt, Bike Fit
For example, if your position is too now and short for your anatomy you may compensate in various ways, one way can be go flex your thoracic spine such that you start to affect breathing mechanics. If this goes on too long your body can start to permanently change your breathing patterns, where you start to rely on secondary muscles to breath, rather than the primary muscles. These muscles will fatigue more easily which could lead to cramping and the inability to breath under heavy efforts.
You suggested your fit was good “on paper”. I would also be careful of trusting "standardized" fits, especially fits based only on body height. Everyone's body proportions are quite different. For example, if you have relatively long legs, you will have a relative short upper body for your height. The longer inseam effectively reduces any frames stack height and while the shorter upper body makes the frame reach effectively longer. Upper to lower arm/leg proportions also vary by up to 30% in people which can further affect fit. It is easier to ride a lower stack height with longer upper arms, while longer lower arms extend the reach and requires a higher stack height in a road bike setup).
Personally, I have a long inseam and long lower arms, this requires a much taller stack and longer reach than would be expected for my height. Prior to that I often rode a position that had too low of a stack height and too short of a reach, this caused me to compensate by excessively rounding my back in the thoracic region, which started to impact my breathing mechanics slowly over time. At one point as I would inhale my diaphram would only come part way down, then lock in place followed by breathing "up" (i.e., apical breathing). I ended up getting muscle spasms in my intercostals and at the top of my rectus abdominis right near the rib cage. The solution was to raise my stack by a 1-2 inches (due to having about 2 inches longer inseam) and extending my reach by over an inch for my longer arms. This ended up straightening my back more which opened up my breathing greatly. I also had to work on gaining back thoracic flexibility and retrain my breathing mechanics. My final position did not differ much in aerodynamics because I am more stretched out.
You may wish to get assessed by a professional as you may be unwittingly compensating for less than optimal fit by impacting your breathing mechanics.