The following forces are acting on your bike as you roll downhill (in more or less decreasing order):
The force of gravity is proportional to mass. If two riders have the same aerodynamic profile, the heavier rider will descend with a higher maximum speed.
This is easily seen in the formula for terminal velocity. The intuitive explanation is that the two will experience the same force due to drag, but the heavier rider will have a higher force due to gravity.
As you are heavier than your friend, gravity can't explain why he is faster.
Drag has a huge effect on descending speed. It's possible that your friend is getting into a tighter tuck, which as @ChefFlambe pointed out can even include your leg position. The aerodynamic properties of the bicycle (the frame, rims, spokes, etc.) can also make a difference.
My guess is that rider position is the most likely explanation for your difference in speed. Since you are taller than your friend, it may be harder for you to get into a tight tuck. You mentioned that you think it might be the rims. If this is the case, the front wheel will make a bigger difference than the rear wheel, so you could try swapping front wheels to rule this out.
A cyclist who continues to pedal downhill will go faster than one who is coasting. A bicycle with a higher top gear can be pedaled at a higher speed, and a cyclist who can ride at a higher cadence may have an advantage.
To rule out pedaling as the explanation, try going downhill with both of you coasting.
Tires with high rolling resistance are slower than tires with low rolling resistance. Factors include not only tread type and width, but also the type of rubber and casing.
If you both have similar 700x23c wheels at roughly the same pressure (or slightly higher for the heavier rider), then you can rule this out.
Friction in the hubs will slow down a bike, but this force is probably insignificant unless there is a problem with bearings.
Another source of friction is in the freehub or freewheel, which will be ratcheting (clicking) while coasting. The friction may be difficult to compare against your friend's, however, unless you remove a lot from each of your rear hubs.