I've experienced this problem when the small chainring has one lose bolt, so check for that, and also that the chainring is perfectly flat. Note that prolonged use of a chainring with lose or missing bolts may result in bending it.
A wobbly chainring wont have a uniform distance from frame's centerline, thus, it won't have a regular position relative to the front derailleur. In my cases, the same can happen between large and middle chainring too, but is perceived only as "difficult shifting" rather than chaindrop.
This could explain why it happens only some times, because you wont initiate shifts always in the same rotational position of the crank. (e.g. sometimes you shift when right foot is up, some times the opposite)
Another cause for wobble could be a lose crankset or a faulty bottom bracket, although these two are more rare. I've experienced both problems on, admittedly, extremely neglected bikes. A loose crankset on a square taper spindle, that luckily was properly tightened before permanent damage was done, and a poorly sealed BB cartridge that ended in catastrophic failure (should have replaced it way sooner).