It should not be surprising that it is difficult to ride in a straight line with both hands off the handlebars when there is a crosswind.
However, I find it most unintuitive, that for example if there is a crosswind from the left, if I let go of the bars and try to keep my body upright, the bike wants to turn sharply to the left (into the wind). To counter this, I have to lean my body to the right, as if wanting to turn right.
I would have thought that the wind pushes me to the right and I have to steer left to keep going straight. Why is this not the case?