I find that any hole, cut or tear larger than a pinprick is impossible to repair with normal patches because they are too soft. Even a 2-3mm slit will cause the repair patch to balloon out and burst at that point. Is there a way around this?
I don't think that you will have a problem with the patch ballooning out as long as the tube is inflated while fully mounted with rim and tire. To my experience there just won't be any room to balloon to.
If you fear that this will be a safety risk, I would recommend a new inner tube when the old one has such a big hole. When you are on a ride and have no spare tube at hand, even with a 2-3 mm slit a standard patch should be enough to get you home.
1) Tubes are pretty cheap. Better to replace one with a large hole rather than attempt to patch it. (Though 2-3mm cut should be easily repairable with a standard quality patch.)
2) If you have trouble with the tube "ballooning out" then it's not being properly supported by the tire. You must have a cut in the tire that (if it doesn't render the tire damaged beyond repair) requires a durable "boot". A tube is intended to be completely supported/contained by the tire and should not "balloon out" even if infinitely thin and pliable.
First of all: do not bother trying to patch too big holes. If it is just too big, just trash the tube. It is cheap enough and it's better than struggling and trying 10 patchings, wasting a whole patch kit all just to end up with it leaking and having a flat tire in the middle of nowhere.
Now, I have already successfully patched a hole as big as 5 mm. It was that big that the tube could not even be inflated to find the leak, it was not a leak, it was just an opening...
For that I used of of the bigger patches that are included in most kits. That worked just fine.
Apart from that I learned that my biggest mistake of patching over time was not to let the cement dry long enough. 5 or 7 minutes seems about fine before applying the patch. And do not hesitate to put it on a large surface to make 100% sure 100% of the patch surface is on a treated surface. This is actually easy to overlook especially on narrower tubes (such as those for 23mm and less tyres). And when applying the patch, make sure to press it hard on the tube, and especially the edges: these are the critical parts that you want the most to stick to the tube since they will be the first to "suffer" when facing an inflation, and if an edge loses adherence, it is the whole patch that is compromised.
I also had the feeling that the "feather" edges attach better handle better the expansion when inflated than the straight edges.
Nobody's said it explicitly, so I will: ordinary patches are completely fine for slits and holes a couple of millimetres across.
Your description of the patches "ballooning" suggests one of two things. Either you're inflating the tube to moderately high pressure outside the tyre, or there's a significant hole in the tyre that the tube is coming through. In the first case, the fix is "Well don't do that, then". Once the patch glue has dried, you can inflate the tube until it just starts to expand, to check that there aren't any other holes, but you don't need to go any farther than that. In the second case, you need to repair the tyre, too.
Having said that, I'm a bit confused by anything ballooning at all. Patches are usually less stretchy than the tube itself so, when you pump up a patched tube outside a tyre, it's usually narrower at the patched area, not wider.
Have you try using roofing patch? Try cut a giant patch from old tube.