So, I need to be able to carry 3-4 days worth of groceries for one person, about a mile from the store to home. I'm not too keen on the idea of panniers or anything else that mounts to the frame. I don't want any extra junk burdening me during the 99% of rides that aren't grocery runs. So, I've been wearing a backpack but its a little small and not well suited to carrying groceries. Does anybody have a suggestion that will work in my use case?
Get an ortleib messenger backpack classic. It will hold a ton of stuff and it's waterproof in case something bursts inside (it will stay inside).
You may also want to consider a quick release handlebar basket, depending on whether you have drop bars or not.
I don't want any extra junk burdening me during the 99% of rides that aren't grocery runs.
I use a Bontrager pannier (or two) like this one: http://bontrager.com/model/08102
The rack on my bike is permanent, but you can clip the bag onto onto the rack, or detach it again, in a second or two: so put it on the bike only when you're going to the store. I also take the bag into the store so I can fill it there, then bring it out already full and attach it to the bike and cycle off.
If you are carrying much weight panniers are much easier. The bike balances better and they don't put strain on your back like a heavy rucksack does.
I don't think that there is a proper rack that you can fit if you have rear suspension on your bike. I wouldn't trust the ones that just bolt round the seat tube with very much weight.
I would suggest looking into Mission Workshop. I have two of their bags, and they are great quality, made in America and lifetime warrantied.
I agree with the other answers suggesting a pannier - panniers are more suitable for transporting stuff than a backpack.
However, if you do not transport anything except for short shopping trips, you can just use any half-decent backpack. Groceries for 3-4 days is not really that much, and it's only for a mile, so ergonomics should not be such a big problem.
There are many special bike backpacks, but they are usually designed for bicycle tours, to hold a helmet, a bottle etc.. For transporting groceries I'd just use a regular backpack.
I would go with pannier and a quick release rack. Something like http://www.topeak.com/products/Racks/MTXBeamRackE-Type
I fully understand your point about buying a rack and panniers set, possibly involving a new frame/bicycle, so the following is just a suggestion for the long run.
In the long run, it's no good for your back to carry heavy weights, especially when cycling as your back position is not optimal for carrying weights (depending also on how much do you lean on the handlebars).
Also (touch wood) a fall with a heavy backpack is in principle (more cyclist mass=less acceleration) better, but in practice much worse than a fall with panniers, when the extra weight rests on the bicycle. Also, a backpack won't allow you to roll on the ground, happily dissipating excessive momentum during a fall (this is solely based on my own experience though, somebody might want to ask a dedicated question on this topic :) ).
I use those sackpacks:
In practice, they work very well. And when not in use, you can fold it up and tie it to your top bar or something.
They are also very washable. I sweat a lot when biking and I just simply throw it in with my laundry and comes out clean and dries very easily.
Here's a genetic answer for you: One that neither comes up too high nor is too wide at the top. My camera rucksack is reasonably large, but because it narrows significantly at the top I can look over my shoulder with no problem. The backpack I'm more likely to use for riding to the shops is not much bigger, but makes it much harder to look behind me (though it is a better shape for carrying (e.g.) bread at the top).
I suggest that you might like chest and waist straps to keep the bag tightly attached to you, and possibly compression straps to shrink the bag to the load. These should all stop the bag swinging around.
Proper loading is important as I mentioned in my comment - keep the heavy stuff low for stability, with the benefit of not squashing other shopping.