Yes, there is an efficiency gain by turning the lights off.
Cyclingabout made a hub dynamo comparison based on an experiment by skjegg, where power consumption was measured for different dynamos attached to different lights.
To give you ballpark figures, let's use the example of the Busch & Muller IQ-X light. The first graphic shows pedaling drag (dashed line) and output electrical power (solid line) for when the light is on.
The second graphic shows pedaling drag with lights disconnected, which is similar to turning the light off.
By having an hub dynamo installed, you constantly loose around 1.5 - 5 W (at speeds above 15 km/h - lights off scenario). However, depending on light and dynamo, by keeping the lights off you can save around 8 - 11 watts.
An important follow-up question would be: does it matter in practice?
According to bikecalculator, for a 100 kg rider + bike system, flat, no wind, one needs about 115 W to cruise at 25 km/h and 39 W to ride at 15 km/h. Let's see how faster one could ride if this extra drag were to be converted into forward movement. I will use data from the Shimano 3D32 (the one I see most often) and assume that the calculator does not take into account hub dynamos.
15 km/h (39 W):
- +2.5 W (41.5 W total effort - hub dynamo & lights off) would equal 15.5 km/h, i.e. approx. 0.5 km/h faster;
- +13.0 W (52 W total effort - hub dynamo & lights on) would equal 17.42 km/h, i.e. approx. 2.4 km/h faster.
25 km/h (115 W):
- +3.5 W (118.5 W total effort - hub dynamo & lights off) would equal 25.24 km/h, i.e. approx. 0.2 km/h faster;
- +10.5 W (125.5 W total effort - hub dynamo & lights on) would equal 25.85 km/h, i.e. approx. 0.8 km/h faster.
Note that these values are for instant speed. The average speed difference would be smaller.
EDIT: I originally made a mistake on interpreting the graphic that skewed the results significantly to the draggier side, thanks @jpa.