In seeking to just replace the rear derailleur (RD), which is the least expensive option, you would just need to find a replacement RD that works with the same number of gears you currently have with the Sora groupset. For example, if your current bike has a 9-speed cassette on the rear (this is what is spec'd on the 2016 Defy 3 for example), you could replace it with the 105 equivalent: RD-5500-GS (long cage) or the RD-5500-SS (short cage) rear derailleur. The long cage version can handle a larger range of gearing (larger range in the cassette and the front chainrings combined).
However, it may be challenging to find a new RD-5500 around. Not impossible, but there has been a couple of generations go by. This is assuming that you are currently on a 9-speed cassette.
The next step up in cost involves what you mentioned at the end of your question: replacing the RD, cassette, chain and rear shifter; and would it be compatible with the current crankset and FD. The general rule of thumb is that a one gear change (say 9 to 10 speeds) on the rear can work pretty well with the current FD/chainrings. It starts to get a bit chancier for seamless operation after that. This is due to the narrower chain width and the spacing between the front chainrings among other things. It does not mean it will not work, but it gets riskier, so it is best to be aware of the possibilities of an upgrade on the FD/chainring area.
You may find that moving to a 105 groupset at the 11-speed level may be prudent at this time. 12-speed 105 just came out and it is electronic shifting only as I understand it. So procuring a full 105 groupset (maybe if possible, without the brake calipers) could be an option, and eliminate potential issues with the FD/chainrings with a hybrid configuration. I have seen new full 105 7000 series groups on eBay for $700+, but that includes everything.
One other KEY consideration to going to 11 speed is this: an 8/9/10 speed freewheel (likely what you have if you are not already on 11 speed), will not take an 11 speed cassette (only one exception and that is the 11-34 11-speed cassette which will fit an 8/9/10 freewheel). So going to 11 speed in the rear could add the additional cost of a new wheelset, or at least a new rear wheel.