While in your question you say you don't use a chamois, I'm going to give some general recommendations for others looking to find quick dry shorts or pants. Since specific recommendations go out of date, I'm not linking to any particular brand or model of shorts...
Almost every pair of mountain biking shorts I have owned would be what I would consider "quick drying". Baggy mountain bike shorts usually have two main components, the inner padded liner, and the outer shell. Here is what I would look for:
Removable liner - If the shorts get so wet that the inner liner is soaked through the outer shell is going to take a long time to dry, and you are going to be uncomfortable. With a removable liner you can swap out to your normal underwear when you get to your destination.
Nylon/Polyester shell - For the shell to be quick drying you are going to have to manage the trade off of fabric weight and durability. A lighter but rip-stop type fabric provides a good mix of quick drying and durability against rips and abrasions. Watch for stretchy panels or cotton fabric pockets as these will stay damp much longer.
An option if you need to wear pants is to buy pants from the same kind of nylon material and wear standard or liner-specific bike shorts underneath them. Most major outdoor manufactures like North Face, Ex Oficio, Patagonia, Columbia, etc, make pants and short from quick drying nylon. There are many online bike clothing suppliers like REI, BikeNashbar and Performance Bike that have house brand baggy shorts that would work as well.
A key thing to avoid if you are really worried about comfort and drying time is pants/shorts with the liner sewn in. With those, if the liner gets soaked you are stuck. If the shell or pants have a removable liner, you can always take the separate pieces off and wring them out to speed drying if they are really wet.
Finally, if it's a short enough ride, or your backside is ok with it, just use regular nylon pants or shorts like above with your normal underwear. I usually only worry about using padded bike shorts if my ride is going to be longer than 6 mi / 10 km. or so. And make sure your bike has fenders. Those make a HUGE difference in how wet you get in the rain.