Depends on how "little" the accident was.
First double-check that the handlebar really is "square" to the fork, and not slightly cocked one direction or the other. (Though this problem shouldn't cause the wheel to turn when you let go.)
Next, oddly enough, do the same check with your seat. If the seat is slightly angled to one side or the other then it will put your weight off-balance and you will get the tendency to turn. (Also, note that you'll have this problem if you put weight in a basket on one side of the bike or some such.)
If not those, pick the front end of the bike up and spin the wheel -- make sure it doesn't wobble at all as it turns. Do the same with the rear.
If not those, I'd suspect that the frame is bent. If you stand in front of the bike and imagine a line going down through the center of the steering shaft, that line should be exactly in line with the frame of the bike behind. If it twists to one side or the other then that's your problem -- bent frame.
Also look at the fork itself (a slightly more difficult thing to do accurately). In your accident one "tine" of the fork may have gotten pushed back relative to the other. Especially likely if the accident involved hitting something.
In any event I'd suggest you take the bike to a bike shop (or a knowledgeable friend) and have it checked out.