Rear derailleur lever geometry, shifter cable pull and clicks amount control rear derailleur lateral movement amount. Let's assume that there are 11 cogs with distance of 10mm between each other and derailler moves 10mm every click. 11 speed cassette might have the same width, which means 10% lesser distance between cogs, so it's 9mm. When otherwise 10 speed configuration would shift over such an 11 speed cassette, some gears are going to shift kinda successfully because 10% over one step might not introduce much of positioning error. But over ten shifts the error would be equals to one whole speed, so issues are bound to happen somewhere. I've used non-real-life values for example purpose. These differ between manufacturers and individual part models.
The said above also means that it might be possible to find components combination that allow you to use rear derailleur and cassette designed for different amount of speeds. Some real life examples:
- SRAM mtb 9 speed shifter makes Shimano 10 speed mtb rear derailleurs compatible with 9 speed cassettes. There's no shadow+ for 9 speed, so some people choose this option because of price or weight.
- Shimano 10 and 11 speed rear mtb derailleurs are compatible.
- It's possible to modify existing derailleur so it moves less or more. Some people has modified 10 speed Shimano mtb derailleurs to work with otherwise 9 speed Shimano 9 speed drivetrain. Basically, they've routed the cable through hole drilled slightly closer to the cage, which changed lever geometry. I think the same can be done to increase the lever length with some spacers.
- The shifter indexing cog can be replaced. There are some companies that shift such cogs that transform 10 speed shifter into 11 speed. DIYing cog should be possible too.
In your case, without modifications and assuming you are going to run same manufacturer components, 11 cogs would cause poor shifting with jams or skips. I'd rather not do that, but if you are mechanically talented, you can always try luck with existing gear mods.