No, there is no functional downside in the sorts of cases you're talking about, given that you can find a good resting angle for the lever.
The world is in flux on the decorum versus functionality aspects of this issue. You can see this most clearly in how major manufacturers have diverged in which side of their thru-axle forks to put the threads on over the past few years.
Designs where the cam lever is offset from the axle, such as most internal cam QRs, often are tricky to get positioned well on the drive side, without it sticking off into space too much. Running it sticking off into space introduces hypothetical risks of it getting bonked open accidentally that would need to be weighed against the hypothetical risks of the issues you mention of having it on the rotor side. Designs where the lever is centered over the axle usually make it easier to find a good path on the drive side, i.e. pointed between the chainstay and seatstay.
Non-mechanical users can have trouble in the best circumstances finding a resting angle for their QRs that allows it to fully close instead of bottoming against the frame/fork. Introducing other obstacles to that isn't good because the safety considerations are more potentially real when done wrong, and probably in practice trump any other considerations. It's for this reason I never put the rear QR lever on the drive side as a working mechanic. But I do put the tool interface side of non-QR skewers on the drive side of disc bikes if I'm making that choice, because then there's no downside.
Given the above issues with drive-siding the rear, whether to non-rotor-side the front is usually really a choice about whether you're going to let a bike out with mismatched lever sidedness. This is of course a weird looking break in decorum to anyone who hasn't thought about all this. My answer here is it's technically better if you can get a resting angle you like on the lever, which is most of the time, but I spend enough time being the sad freak taken technically driven stands about other corner case issues in wrenching that I can't bring myself to get too invested in this one, which is what I would be doing if I had the skewers all funny on every other bike I touched. Such goes cultural change, which is really what's in question with front skewer sidedness.
In the back on thru-axle bikes, having it on the left provides the advantage that you can put the threads in the hanger. That consideration probably wins against all others there, so the question going forward as it relates to thru axles is whether manufactuerers are willing to make forks opposite in pursuit of the slight practical edge at the cost of adherence to decorum. I've been surprised to see any have answered yes to that, but they have, so who knows where we're going.