A bearing is an interface that enables the axle to rotate inside some fixed part. The axle rolls on the bearings. For example, consider your hubs, pedals, the cranks, and your front fork.
Sheldon Brown has a longer discussion on bearings. Most bearings are steel balls.
Preload is important term related to bearings. For your hubs, preload is how much the hub is getting squeezed in from the side, ignoring the quick release skewer. All cup and cone hubs have inherent preload adjustment, but not all cartridge bearing hubs or other devices do. Too much preload will make the bearings grind and wear out. Too little preload can leave play in the hub (i.e. side to side movement when the quick release skewer is clamped) or other device (e.g. crankset, headset will rock back and forth).
Cup and cone bearings
This type of bearing, pictured above, has the balls rotating between the cup and the cone. The locknut tightens the assembly together, and can be loosened with a standard sized wrench.
Cup and cone bearings require periodic service. Service involves cleaning out the old grease, which may have had some dirt mixed in with it, and adding fresh grease. If the balls are worn, you can measure their size with a caliper and buy replacements. Manufacturers may make the cones available as replacement parts as well. The cups may or may not be replaceable.
Cup and cone bearings are more traditional and are frequently present on older equipment, but some companies persist in making high-quality cup and cone bearings (notably Shimano and Campagnolo).
Cup and cone hubs all require preload adjustment. Best practice is to tighten the locknuts so that there is barely perceptible play when the hub is off the bike, and zero play when the quick release or thru axle is tightened.
Most manufacturers have moved to cartridge bearings. Here, the entire system, comprising the steel or ceramic balls, the seals, and the inner and outer races (roughly equivalent to the cone and the cup respectively) are manufactured as a unit. Sometimes they are called sealed bearings, and most of the bearings in bicycles have rubber seals, but this is potentially a misnomer. Bicycle bearings might be unshielded at one end, and cup and cone bearings can have seals built in.
To replace these, you will need a bearing extractor to pull or push the bearing out. The bearings themselves are available commercially. All cartridge bearings have a designation. The numeric part of the designation will specify an inner and outer diameter plus a depth. The bearing code is usually printed on the rubber seal. The numeric part of the code identifies the dimensions. There may be alphabetical suffixes providing further information, e.g. about the sealing arrangement. Do note that bearings with less-protective metal shields exist, as well as unshielded bearings. Those should generally be avoided in bicycle applications.
Most cartridge bearings are designed not to be serviced. They are replaced entirely. In principle, it may be possible to carefully remove the seals with a penknife or similar implement, then clean out the grease and add new grease. Some higher-end manufacturers (e.g. Chris King, Kogel) design their bearings to be serviced periodically, but they may also supply replacement seals if you damage the original ones during removal.
Many cartridge bearing hubs do not have a preload adjustment mechanism, so there is no need to worry about preload. However, some do. SRAM cranks require you to use a collar to preload the system. For Shimano cranks, you just tighten the crank cap to finger tightness, then tighten the pinch bolts. A similar process applies to headsets.
Many factors may influence the durability of the bearing. If you ride in wet conditions, this will reduce durability. If you pressure wash your bike, take care not to point the hose straight at the bearings (in your hubs, bottom bracket, rear suspension pivot). Some bearings are more tightly sealed than others, but the tradeoff is a small increase in friction.
In a press fit bottom bracket, if the bottom bracket seats are not parallel to each other or if one is out of round, that can damage the BB over time. If a piece of equipment was designed with bearings that are too small to take the loads applied to it, that would also impact bearing life. Last, the material quality of bearings and the races they run on has an impact.
Ceramic bearings are sometimes offered as an upgrade on high-end bikes. They have marginally lower rolling resistance than high-quality steel ball bearings. In many industrial applications, ceramic bearings are thought to be more durable, but this may or may not transfer to bicycling applications. In particular, ceramic bearings are extremely hard, and they must be paired with a high-quality steel race, or else they can damage the race when the bike goes over bumps. This would argue for avoiding cheap ceramic bearings.
Bushings (link to Wikipedia) are another form of bearing. These involve a sleeve rotating over the axle. Bushings are often used in rear derailer jockey wheels and in mountain bike suspension.