The most critical thing is the "reach" -- how far forward and up/down the handlebars are from the seat, relative to your torso and arm length and, shall we say, avoirdupois. Too short a reach and the torso must curl up, compressing the lungs and causing strain, and the ability to assume an aerodynamic crouch (if you would wish to do so) is reduced.
On the other hand, too long a reach places a strain on the arms and back and significantly increases the pressure on the hands.
Moving the seat back to increase "reach" makes the angle between torso and legs more acute. Up to a point this moves one closer to the "sweet spot" of maximum power from the legs, but this can go too far, plus the problem with an over-ample gut compressing the lungs is increased. Moving the seat forward generally moves one away from the "sweet spot" and also increases the weight on the hands and arms.
Moving the handlebar forward/back to adjust "reach" upsets steering stability/agility.
A too-small bike may (in rare cases) not allow the seat to be raised high enough to attain full leg extension, or (more often) not allow the handlebar to be high enough for comfort. A too-large bike may not allow the seat to go low enough, forcing the cyclist to slide side-to-side as he pedals.