I have a set of Xero Lite XSR-3 road wheels with 20/24 straight spokes. I tend to ride dirt, gravel, and some singletrack on my cross bike and sometimes some cross races. Would I be able to swap this wheelset on to my cross bike to save weight or should I stick with my heavier duty 32 spoke wheels.
They should be fine. There are a lot of people that ride "road" wheels for cyclocross. Unless you have some dedicated tubular wheels with wider and shallower glue beds to accommodate cross tires, you're probably riding "road" wheels with cross tires on them anyway.
I'm currently using some Mavic ksyriums on my cross bike as training wheels and they work out just dandy. Make sure you check the spoke tension often as all the jostling and off road riding will tend to loosen the spokes and throw your wheel a bit out of true quicker than road riding.