I regularly cycle barefoot for short distances during summer or when it's raining and I don't want to get my shoes wet. (Note: I live near the coast, where it never gets cold enough to make frostbite a significant risk.) I've dealt with the spikes on my pedals by taping foam around the pedals, so now cycling barefoot is perfectly comfortable. I haven't done it for long distances, though, so I can't comment on whether it stays comfortable.
The only injury I've gotten in a few years of doing this happened when my chain slipped and I drove my big toe straight onto the tar. That hurt. I don't have experience with other injuries from cycling barefoot, but the risks other posters have pointed out (like stepping on something sharp, getting sunburnt or having your feet badly scratched) also seem real and fairly probable. One more thing to consider is that if your brakes fail and you have to stop using your feet, you're going to wish you were wearing shoes. Ouch.
I've chosen to take the above risks because of the comfort and convenience that cycling barefoot brings. I no longer walk around wearing wet shoes all day, I no longer have to deal with flip-flops falling off or tripping me, and cycling in summer feels much more pleasant. I acknowledge that stubbing my toe into the road would have caused much less damage if I'd been wearing closed shoes, but I still think that all-in-all the decision to cycle barefoot has been worth it for me.
The biggest drawback, from my perspective, has been social. As you've pointed out, cycling barefoot is unusual, so people question it and sometimes make fun of me. That's unpleasant, but again, I've chosen to do it anyway.