Once again, I spent some quality time debugging a slow leak and found out it was a loose valve core. I tightened it with pliers, but it raises the question, is there a situation where it is useful to be able to remove the valve core? Also, have replacement valve cores been available at some point, and do the cores have a standard thread or is it manufacturer-specific?
As others have mentioned one reason is to make it easier to installed slime/sealant into a tube or tire. Typically you'll take the valve off, and use an injector to put the slime into the tire.
For tubeless tire setups, removing the valve core allows a higher volume of air to quickly get into the tire when you are first blowing it onto the rim.
Some types of valve extenders used by people with deep profile rims thread into the valve core threads.
Another reason to have removable Presta valves is they tend to be somewhat fragile so being able to replace a busted valve vs. having to replace the whole tube could potentially save a little bit of money (or allot of money if we are talking about breaking the valve core on a $150 tubular tire.)
The valve cores should be consistent from maker to maker.
I think the big benefit of a removable core is that it makes it possible to add "self-patching slime" to the tube. With a core in place the slime would likely foul the valve and make it inoperable. Otherwise, I think the removability is more an artifact of the manufacturing process – and as you've noted, not necessarily a feature.
The accepted answer briefly mentions deep carbon wheels. I will elaborate.
50-60mm deep wheel are common in many road races, and triathletes and time trialists will go as deep as they can, probably all the way up to 90mm front and a full disc wheel rear. I believe many valve stems for performance road tubes are around 48mm in total length. This won't offer enough valve on even a 40mm deep wheel. Tubes with longer stems are available, e.g. Vittoria makes a 60mm valve stem. However, even this isn't enough for the deepest wheels, and manufacturers would have to maintain more SKUs (stock keeping units, i.e. different models) of their high end road tubes.
Valve extenders are a solution. Two versions exist: one where you just open the presta valve, screw the extender on, and leave it, and another where you remove the core, insert it into the extender, and then screw the extender on the original presta valve.
Having a valve extender can also simplify your spare parts logistics. Many riders may train on one set of wheels, and race or do harder rides on deeper race wheels. Those who do this would either need to carry a spare tube with a valve suited to their deepest wheel, or just carry a valve extender of suitable length.