How can I tell if a rim is bent or if the wheel is just out of true?
The common qualification here is an important one: truing is a skill that benefits from practice and experience (chicken, meet egg). Your wheels are important, and it's important to have somebody who knows what they're doing at least check your work if you're just learning (or if you're disinclined to trust your well-being in dense traffic to wheels you trued yourself).
Having said that, a rim that isn't bent should have more or less even spoke tension across all the spokes when it is true. On the other hand, a bent rim will require higher tension in some spots and lower tension in others. So, noticeably uneven spoke tension in a true wheel is one sign that the rim might be bent.
If you remove the tire and loosen all spokes completely, or better remove them as well, and put the rim on a flat surface, it should become clear if the rim is bent or not. Obviously this requires some work which may be unnecessary if the wheel simply needs truing, so I'd try that first. If it fails and you are thinking of replacing or repairing the rim, it will need to be removed anyway.
Well, sometimes the rim is visibly bent, especially if there are creases in the sides. But if not visibly bent, and you don't have an experienced eye, then you have to attempt to true the wheel and see if it trues up well.
Some minor dents in a rim can be repaired with the proper equipment, but others require rim replacement.
Beyond that you need to read a book.
(It should be noted that if the wheel is not particularly expensive then this is a good opportunity to learn wheel truing. If you're reasonably careful (and use the right spoke wrench) you won't screw it up much worse, and, unless you true it, you're going to have to take it to the shop anyway. So a good time to give wheel truing a try.)
And in response to @WTHarper:
It helps to know how the wheel got cocked up in the first place. If it was hit by a car then the odds of a bent rim are pretty high. If it "just happened" -- the wheel went out of alignment on its own -- then a bent rim is very unlikely (and a broken spoke is likely). If it happened in a "soft" fall -- eg, the wheel slid out from under you on loose gravel -- then a bent rim is possible but unlikely. It's more likely that some spokes were just stretched out. If it happened in a hard fall or when hitting an obstruction then a bent rim is more likely but far from certain.
(And I should have mentioned checking for broken spokes: First check that all spokes are tight, and if any are loose check whether they are in fact broken. It would be very unusual for a wheel to not "taco" slightly when a spoke breaks, and replacing the spoke should get it back in true with relatively little effort.)
The best advice here is to take your wheel to an accomplished wheel builder who should be able to true your wheel if it's possible or rebuild it if it needs it.
Truing wheels is one of those things that's more Art than Science and once it starts going wrong, it just gets worse.
First: don't try truing a wheel you rely on without experience!
If spoke tension is high on the side of the rim deviation, then you can improve the true by adjusting spoke tension. If spoke tension is low on the side of the deviation or the tensions are roughly even, then the rim is bent and trying to fix it with tension will make the tension balance worse.
If you can hear tones pretty well, you can check tension by plucking spokes. Either pluck at the cross (in which case be aware you're conflating the tension of two spokes) or damp the spokes at the cross and pluck one spoke rimward of the cross.
On undished wheels, you can compare the tensions of spokes on the two sides (i.e., coming from different flanges of the hub) directly. If the wheel is dished (as with geared rear or most disc-brake wheels), you'll want to pay attention to the relative tension with respect to same-side spokes, and compare that between spokes on both sides of the wheel near the deviation.
For example, if I have a road rear wheel with a rim deviation to the left (non-drive), I know the right-flange (drive) spokes are at higher tension than the left, so I can't compare tones directly. I'll pluck some left-flange spokes near the deviation, and then some other left-flange spokes around the wheel, and do likewise for right-flange spokes. If left-flange spokes at the rim deviation are tighter than average, and right-flange spokes are looser than average, then I can true with a spoke wrench. If it's the opposite, or if the sides are the same, then it's a judgement call (regarding whether truing will be able to correct the deviation while maintaining acceptably even tension).
The cause of your question should probably determine if the rim is bent or just needs "adjusting" by a qualified wheel builder. If contact with a vehicle or other solid object is the cause of the question, then there's a good chance the wheel is actually bent and will likely need to be entirely re-built or replaced ( replacing may be more cost effective ). If you just discovered the wheel seems to be "wobbling" in a certain spot or is rubbing the frame or forks in a certain area, the problem is likely a wheel rim that needs trueing by spoke adjustment or spoke replacement in a certain area of the wheel rim. In either case, in the interest of safety and rim longevity, I'd suggest having a knowledgeable wheel builder at a bicycle shop take a look at the wheel. That's better than worrying about a possible accident caused by a bad wheel.