There is not a generic answer to this question, and the reason is that handlebar bags are the best in many regards, especially for day rides and light touring, but they do change the handling of the bike, as does anything that puts weight on the fork or bars. They work best on bikes that have front end geometry designed for them, specifically a low trail number.
This is a large subject, but a simple way of thinking about it is that loads in front of the steering axis slow down the handling of the bike. Low trail numbers speed up the handling of the bike. Combining a low trail front end with a front load can be made to result in handling that's neutral, allows easy corrections in cornering, and is easy to handle while fatigued. Unloaded, such bikes are manageable but a little squirrely, and that is the only real disadvantage of the concept. A front load on a bike not intended for one can cause the handling to become slow and laborious to the point that controlling the bike is difficult.
The design dynamic of combining a front load with low trail to achieve neutral handling was well understood by many framebuilders and cyclists around the middle of the last century, and then got lost in the hubris of the bicycle industry. It never really made it to North America en masse. It has been undergoing a revival in the past 20 years, but few mainstream companies have yet done anything with it, and it has also not yet penetrated conventional cycling dogma. Below is my bike, a Soma Grand Randonneur frameset, about the closest thing there is to a mainstream offering. I can put 12lbs or 5.5kg in the bag and it will still handle like a road bike.
Loads behind the steering axis of a bicycle create a tail wagging the dog effect and require additional steering inputs from the rider to keep the bike balanced. For a design where this kind of load is expected, the head tube angle is usually slackened, creating a high trail number. This doesn't reduce the need for corrective steering inputs, but it does make them more manageable for the rider by slowing down the handling of the bike. The disadvantage is you then have a bike that handles slowly in all circumstances, and one where the rider must contend with increased wheel flop.
Contemporary gravel, allroad, and touring type bikes from mainstream companies virtually all follow the latter design. They have fork offsets in the 45-50mm range and head tube angles of 70-72°. They are mid or high trail. They want to carry most of their load in frame bags, seat bags, and/or panniers. Of these, the lower and more central you can get your center of mass, the better the bike will tend to handle. A smaller handlebar bag can still be used but it's best to keep the load under 5lbs or 2.25kg. Handling impacts will be limited with this kind of front load and it's still enough to carry the frequently used items for a day ride, like phone, keys, map, snacks, arm and leg warmers, light jacket, etc. You can use a shoulder strap for off the bike. You may be able to notice it in the bike's handling a little, but it's worth not having to dig for these items in a less convenient bag. The same rule of thumb of about 5lbs max in a handlebar bag is also true for conventional road bikes with a 43-45mm fork offset and 72.5-73.5° head tube.
There's an argument that on a bike with mainstream geometry, bento boxes do most of what handlebar bags do without impacting handling the same way. That's valid but they have pros and cons. A major negative of bento boxes is the lack of a see-through map pocket, a feature seen as indispensable by many.
If by day pack you mean you'd like to be able to get off the bike and have a proper backpack to hike with, that's more the purview of a convertible pannier. Panniers are out of fashion for some fairly valid reasons, the major ones being the base weight penalty of their racks and their suboptimal weight placement, but there are nice things about them too, and backpack convertibility is one of them.
