I went for a ride this morning and my handlebars were loose. I now realize that it is unsafe to ride like that but my average speed was dramatically lower. Do you think it is because of my loose handlebars?
With loose handlebars, the primary thing which you should be worrying about is your face - its a great way to lead to a nasty crash. Don't ride with loose parts on your bike. Speed is the last thing you should be worrying about.
You likely went slower due to less control of the bike with loose handlebars making your brain hold you back on speed. Other factors such as environment or tired-ness may have come into play, but loose handlebars would make most people ride slowly and carefully [or better yet, walk the bike], avoiding any bumps until they could tighten down the handlebars if they weren't carrying the appropriate tools with them.
When you're riding intensely, you actually end up pulling up on your handlebars, which you can't do with loose handlebars so you can't ride as hard, so you can lose speed at the top end with loose handlebars.