The problem is ambiguity. Sometimes you'll be unfairly judged by cyclists you're trying to pass. It will happen. Sometimes it helps to reduce the sting of the judgement with some perspective on what's going on.
When you’re on a bicycle, cars are death monsters. You spend your time trying to keep distance from them. You wish they would all just disappear. Even when they're trying to be friendly. It’s like you’re walking around in the woods, and suddenly you’re approached by a woodcutter who wants to have a friendly chat with you, but he’s carrying a running chainsaw casually in one hand, and walking really fast. What are his intentions?
Same sort of thing with drivers zooming up to you. People don’t even realize their mistake, because most people have never had it happen to them. Cross-country bicyclists deal with it all the time. (I am one of those.)
People will cruise up within inches of you at a stoplight and ask “where ya goin’?”, expecting an answer while you’re madly trying to determine if they intend to go straight or turn right when the light changes, potentially obliterating your bicycle and ending your life. If you don’t respond, you’re being rude. It sucks for everyone.
People will honk their horns at you as they drive by, startling you as you attempt to balance on a narrow strip of shoulder, and unless you look up into their window and parse their faces and their upraised hands, you’ll never know if they were honking because they want to encourage you and cheer you on, or because they hate the sight of you and intend to plow you into a ditch.
People will slow down to a crawl behind you, stacking up the traffic, even though you’ve given them the entire open lane to pass you by, while you ride over rocks and garbage in the gutter to make yourself as small as possible. They will hover there, and you have no idea if they are displaying chivalry and waiting for a clear oncoming lane so they can veer into it and give you a wide, respectful berth – or if they are malevolently cursing you for having the nerve to ride such a ridiculously slow contraption on their highway, and getting ready to scream at you or throw something at you from an open window as they shoot forward. You have no idea because you can hear them, but you can’t see them. Or if you have a rearview mirror (like I do, and like I recommend every cyclist obtain), you can see them well enough to know there's a car there, but you still can't see their faces.
In all of these situations, people are either trying to be model citizens, or they are planning to seriously endanger you, and you have absolutely no idea which it is, until it happens.
I was driving my car across Nevada once. It was sweltering outside. I passed a cyclist slowly rolling in the other direction and I knew he would be thirsty. I turned around at a pullout and went back the other direction, passed him again, then went another half a mile ahead of him before I slowed down and stopped. Then I shut off the car, got out with the water bottle in my hand, clearly visible, and waited there while he closed the distance. It was a gallon of water and he was grinning ear-to-ear as I handed it to him, and sure enough he drank half of it on the spot, in one go. He'd underestimated the heat of the day and I knew he had another 50 miles to go. We had a great chat about touring and took a few photos together.
I wouldn't have known to approach him that way if I hadn't already had a dozen encounters of my own on long tours. And I have to take the rarity of this knowledge into account, for however long I ride: People just aren't going to know what it's like. It's always going to be ambiguous. They will always scare you without intending to. And you just gotta accept that your life is in their hands and hope they mean well.
So, it's a scary situation, and there aren't very many good ways to reduce that fear. But there are a few.
The number one thing is, when you're interacting with a bicyclist, never honk your horn. If they're new to the road, just the sheer surprise of it might make them accidentally steer into a ditch. With experienced riders, you can sometimes toot your horn after you've passed them, while also waving to show you're not furious, but usually just waving works as well. Believe me, they're got their eyes on you at that point, they'll probably see you wave. And that'll make them smile and feel relieved, which is great. We're all on an adventure together!
The number two thing is, when you see you need to pass them, be assured that they see you and are trying to do what they can to make that pass happen so you can both be rid of each other as soon as possible. With this is mind, try to hang back as far as you can while still keeping your own view of the situation in the oncoming lane. The closer you hover by a cyclist, and the longer the time, the more nervous they become. What's worse, sometimes long-range cyclists attract a lot of attention just because of their gear, and people will hover behind them just out of curiosity, which they find frustrating, especially when there are other cars around to complicate things. But that's not your situation, and it's rare in general.
Do your best to drift back as far from them as you can until the opportunity to pass presents itself. The very fact that you are drifting back - that you were closer and now you're farther - will signal to them that you are taking things seriously and understand their perspective. That said, number three:
Pass as soon as you safely can! Get out from behind them and get on your way as soon as the visibility and the space is adequate. If you're not on a busy road, chances are the cyclist was riding in relative peace and quiet until you appeared, and is eager to get back to that, rather than this constant state of panic. The longer you linger back there the higher his/her panic will rise. Every cyclist, everywhere, would much rather have you ahead of them than behind. All that deadly force is then pointed the other way!
Please employ these tactics. Every cyclist on the road will appreciate it, even if some of them still react badly in the moment. It's sometimes just a no-win situation that way. But believe me, most of us really get it - we're all drivers too (unless you're in a dense urban environment where cars are optional) - and we'll truly appreciate that you hung back, then made your move ASAP. The almost businesslike appearance of this when it's executed well is especially refreshing. Thank you for asking this!