What to look for on a shop test ride:
Fit: Like you said, it's new and might feel awkward, but usually you can narrow out some that are just not as good.
- Brakes: Test out front and rear braking power, bite, etc.
- Drivetrain: How does the shifting feel, is it responsive? Do the cranks flex? Is anything loose (usually can be fixed on the spot)?
- Cockpit: lean back over the rear wheel and see if you can do it comfortably, check to see if the bar width works for you, stem height, etc.
- Suspension: bounce up and down a bit and see how the shock and fork moves. Now go run over some bumps or try to bunny hop and see what if feels like.
- Maintenance: Look over the entire bike and notice whether the wheels have thru-axles or quick releases, if brake mounts are built in, removable dropouts/derailleur hanger, etc.
Look: Do you like what the bike looks like?
Ride: Ride fast, ride slow, turn wide and turn tight. Then lean the bike while riding, lean forward, lean back, lean to the sides. Basically see how the bike moves with you.
Now the majority of these things you can adjust or change. So what you'll really want to look for is frame size and geometry, the "feel" of the bike, which the shop should be able to help you with. They can also help get the suspension set for your weight and riding style at least to start with, that's another adjustment you can make. If you manage to get a really good shop or if they're not busy, you might be able to persuade them switch bars, stem, seat and other bits to see if you can make the bike feel better.
One note, on the maintenance part, I added this because you can buy bikes that are horrendous to maintain (internal gearbox, weird suspension layout) and some are super easy. I wouldn't make it your main factor in choosing, but it's something to look for.
And, not surprisingly, buying a bike is a lot like a car, you have to like it. That means, color, design, and feel; and that's something only you can determine. Some guys like riding a smaller frame because it's more "flickable" and others like a longer bike to have more stability at speed.
Lastly, there's really no substitute for riding the bike you want on a trail. Rentals, demos, and friend's bikes are great for this. Check with your local shops/clubs to see if you can find a bike you'd like to try and maybe meet someone at the trail.