The important thing to realize is, that there is always an optimal tire pressure for each combination of tire, weight and road bumpiness. If the pressure is too low, you will loose too much energy massaging the rubber of your tire, and if the pressure is too high, you will loose too much energy by the tire bouncing off of the bumps in the road surface. The optimum is where tire is squeezed as little as possible by your weight while flexing away the vast majority of bumps so you don't feel them in your hands.
As such, optimum pressure depends heavily on road surface. If you ride on a washed-out compacted sand path with stones sticking out everywhere, you probably need a pressure somewhere between 2 and 4 bars. If you ride a mountainbike trail, pressures below 2 bars look good. However, if you ride on a freshly built road, your optimal pressure is well above 5 bars. For such smooth rides, the pressure limit of your tires is likely lower than the optimal pressure.
As such, if you compare the feel of your different pressure on smooth tarmac, higher pressure will almost always be faster. However, most riding is not done on perfect roads. Some roads will have old tarmac, some roads may have potholes, some routes will have compacted sand (in different states of quality) or even plaster. The typical bike needs to handle all of these well. And on these surfaces, higher pressure does not feel faster at all, it actually tends to feel significantly slower.