I am refurbishing a bike and am deciding how much to clean out the rear derailleur. It's in pretty good shape. But after I take off the chain, I tried spinning the jockey wheels with my fingers (after cleaning grime off the sides), and they only spin as long as I'm touching/pushing them -- they don't spin freely. Should I be concerned enough to take apart the derailleur and clean/re-oil or re-grease the jockey wheels?
If the teeth on the jockey wheels appear to be in good shape then you probably have nothing to worry about. Many jockey wheels use bushings instead of bearings, and either way they are not going to spin for very long just because there's not much weight to them and therefore they don't carry much inertia.
If there is noticeable resistance when you try to turn the jockey wheel- as in it leaves an impression of the tooth on your finger when you try to spin it- then they need service. If you want to do it youself you can take the cage apart and check the bushings/bearings and replace them if needed. Servicing instructions should be available from your derailleur's respective manufacturer along with replacement part numbers. Be sure to use a very small drop of threadlocker on the threads of the bolts that hold the jockey wheels to the derailleur cage when reassembling, and be sure not to overtighten those bolts.
It doesn't sound like this is a problem in your situation, but for completeness I'll mention that you don't want to let the jockey wheels themselves get over-worn, either. As they wear out the teeth may become narrow and pointed or short and blunted, depending on the riding conditions and the condition of the bushings/bearings. Here are a couple of extreme examples: