You should pay attention to both. Ideally, if I wanted to make the strongest wheel possible, I'd build a 36 spoke wheel with a double wall rim with eyelets, and double butted spokes, and lace it 3-cross. All of these components help absorb or redirect impact.
Double wall rims are definitely the biggest part of this. The torsional stiffness of a double wall can definitely keep things rolling straight. I've even had a wheel with two broken spokes that had a very tall double wall rim, that I rode like that for 2 years before replacing those spokes, because the rim wasn't that far out of true. Of course, this was on a fixed gear with no brakes, so brake rub wasn't a problem...
Eyelets reinforce the spoke/rim interface, making it stronger and helping to avoid nipple pull-through.
Spoke count makes a pretty big difference simply because the more spokes, the more impact gets divided between separate spokes. The cross pattern matters because every point two spokes cross, some small part of the energy divides off to the crossed spoke. More than 3-cross can be a problem though, since you start having spokes crossing at the j-bend where they go into the hub, which can cause more problems.
This last one is going to seem a bit counter-intuitive. Double butted spokes are thinner in the middle than they are at the ends. One thing this does is make the spokes a little bit compliant. Whatever forces are acting on them, they can stretch or contract just a little to adjust for. Also, the weak point of straight-gauge spokes is at the J-bend where they enter the hub. Any mechanic will tell you, that's where they break. The double butted spoke adds a perceived weak point, the middle of the spoke, where it's thinner. Since that part is the smallest bit compliant, it can absorb more impact without breaking.