I have two theories that might explain the wobble you're experiencing. Perhaps both have their part in it.
Self stabilizing of the front-wheel (resonance)
Usually bikes are built in a way that stabilizes the front-wheel. The term for it is trail (called "Nachlauf" in the image). It's the horizontal distance between the steering axis and the wheel's axis. On bicycles this is often realized as a slight forward bend of the fork.
So whenever the front wheel gets a bump there is a force trying to put it back to the straight position. Due to that there is bound to be some resonance frequency. With increasing speed and thus stronger bumps some of bumps are bound to be nearly in resonance with the front-wheel and to create your wobble.
Whenever the wheel isn't exactly in the direction of the driving direction it gets some force parallel to it's rotation axis. As every rotating object it reacts with a movement called precession (just like a gyroscope). Also this gets more intense with speed because of the higher angular momentum and the stronger force on the wheel.
So what would help?
- Increase the trail of the front wheel. See this article on the pros and cons of that approach.
- Sit in a more inclined position - thus putting more of your weight on the handle bar.
- Put your hands further apart on the handle bar (increasing your leverage). You can easily experiment with this: Usually the wobble begins at 20-25km/h if you're driving free-hand and at ~30km/h if you put your hands close together.
- Avoid elastic frames. Especially frames with a low trough step (originally for wearing skirts while driving) seem to add to the wobble.