Okay, let me begin with some terminology clarifications:
- Sweep is an angular measurement, not a linear one. For example, your bars may have an upsweep of 5°, not 20mm. This is an important distinction because it affects the directions your bar grips point at in 3D space.
- Backsweep: the angle the grips point backwards at. This is distinct from upsweep since handlebars have a compound bend. It's inaccurate to model them as a simple V.
- Upsweep: the angle the grips point upwards at. Note that this is not the same thing as the bars' rise.
Now, you can more or less model the range of motion of your outstretched arms as a semicircle. Imagine the handlebar has no backsweep; it can be modeled as a tangent line at the apex of the semicircular arc. I will add a diagram when I can, but you might be able to visualize how when spread apart by the handlebar's width, your arms will meet the ends of the handlebar line at an angle (they form an isosceles triangle). Adding some backsweep to the bars makes them follow your arms' range of motion better, since the grips will be closer to tangential with the semicircle. Hence, in my comment I noted that front sweep would be unpleasant because it would make your arms meet the bar at an even less perpendicular angle than a straight bar would.
The same argument more or less applies for justifying upsweep. Since humans generally have torsos, their shoulders are raised above the vertical level of the stem. Hence, your arms meet the handlebars at an angle in the z-direction, which upsweep compensates for. Like before, downsweep would make your wrists have to bend excessively to adjust for the angular misalignment between your arms and the bars.
Overall, I'd argue that yes, only one quadrant is usable. The angles are just nonsensical in the other three. Additionally, for high-end, lightweight bars which may have an asymmetric internal construction, it might be bad for them structurally speaking to be loaded in directions they were not intended to be used in.
