Beyond the question of finickiness of actually using the system with higher cog counts, it's a matter of whether there's enough total cable pull from the shifter in friction mode to cover the range of movement needed given the cassette used and the actuation ratio of the derailer.
Here's an excellent article with the basic data you need to figure it out, snipped:
The only other piece you'd need to know is the actual total cable pull of your SL-BS77 9-speed bar end shifters, which requires measuring since 9*2.5 doesn't work because it's got some phantom movement both before and after its normal range of clicks when in indexed mode. I have them on a bike so I undid the low limit, put it in friction, put a mark on the cable and came up with 24.5mm total movement.
So, the answers:
Shimano 10 with a 10-speed Shimano road RD works because 24.5*1.7=41.65 (the total amount of derailer movement you get) and 3.95*10=39.5 (the minimum total range you need, here a smaller number than the 41.65). Shimano 10 with a 10-speed mountain RD doesn't work because 24.5*1.2=29.4.
Shimano 11 with an 11-speed Shimano road RD doesn't work because 24.5*1.4=34.3 (derailer range movement), which is less than 11*3.9=42.9 (range needed). Shimano 11-speed mountain RDs have an even lower actuation ratio (1.1) so we know that won't work either. So in other words, all the 11-speed shifters pull quite a bit more cable, so this friction shifter and probably many others out there in the world can't really sub in for them, primarily because the actuation ratio the systems run on has been made quite a bit different than in the past, plus the cassette itself is a little wider. You could almost do it by just using an older derailer with the 1.7 ratio, but not quite (24.5*1.7=41.65, and we need 42.9). Hacking the cable anchor to tweak the ratio by that much is probably achievable.
Campy 10 with a 10-speed Campy RD won't work, because 24.5*1.4=36.75, and 4.15*10=41.5. But it will work, barely, with a Shimano RD with the 1.7 ratio.