When riding in the plain the only thing limiting your velocity at a certain level of steady exertion is friction: Rolling and gear friction as well as drag. Interestingly, the force to overcome rolling resistance and friction in your pedals etc. is constant; if you were riding in a vacuum, your legs would need to exert the same force at higher speeds than at lower speeds (once you finished accelerating of course, which needs force as well, as good old Isaac formulated). The energy spent per distance on a bike in a vacuum does not depend on the velocity. Still: Because you are going faster, you cover more distance per time, expending more energy in the same time. That's what physicists call power, measured in Watt.
The power needed to overcome rolling and drive train friction grows linearly with the velocity.
The second kind of friction to overcome is aerodynamic drag. That force grows quadratically with the velocity: If you go twice as fast, you need four times the force. As before though: Because at higher speeds you cover more ground in the same time unit, the energy per time, or power grows with an additional factor:
The power needed to overcome drag grows with the cube of the velocity.
One consequence is that the drag is negligible for small velocities but grows quickly and quickly becomes the dominant factor restricting speed. The diagram below, from Wikipedia, shows that nicely. The velocity is given in m/s. To get km/h, multiply with 3.6, e.g. 4 m/s equals 14.4 km/h.
Note how the power expended for drag (the blue line) is about 20 W at 4 m/s but about 160 W for double the speed at 8 m/s; doubling the velocity needs eight times the power.
From Wikipedia, Theosch, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0, via Wikimedia Commons
A note to your personal observation that going 15 km/h doesn't seem to be easier than going 20. As David said in his answer, there may be a most "natural" speed for each person. I can imagine though that little children would struggle to maintain 20 km/h over larger distances. Maybe check your tachometer for accuracy? A mis-measurement of only 25% results in a mis-estimation of the drag of a factor 2.
But generally, the energy expended by our bodies and the exhaustion felt does not directly correspond well to the work done in the physics sense: For example, carrying a heavy load across a plane doesn't accelerate or lift the mass at all, and still is very exhausting. Even just holding a heavy load is exhausting. Likewise, standing on an incline and balancing a stationary bike is likely exhausting even though, as in the other example, no physical work is performed. Going slow on a bike uses a lot of muscle movement and other "ancillary efforts" to control the body to no effect. At low revolutions much of the effort simply goes into pressing the pedal without effect (like a rocket just strong enough to hover — all the fuel is wasted). Together, and with the fact that at slow-ish speeds the forces are small to begin with, that may well explain your perception.
But from a physics standpoint the case is clear: For low speeds the dominant power drains correlate slowly, linearly, but for higher speeds they rise very quickly, with the third power, which is why we mere mortals never go 40 km/s without some tailwind (and why even 5 km/h tailwind are a godsend at higher speeds).
