I am buying my first 'real' road bike soon and am comparing the Shimano Ultegra vs. the 105.
I have read through their websites and marketing literature. What does the semi-vast price difference do for a cyclist, who isn't riding the TDF?
There's a couple of things here.
First there are the physical properties of the groupsets. As you move up the groupsets, what you're buying into is essentially smoothness and lightness.
But for a recreational rider, you'd basically need the groupset to hit a certain minimum level of quality, and anything beyond that would be lost by the rider. And my guess is that for many of us the minimum level of quality is probably even lower than 105 - Tiagra and these days maybe even Sora.
In terms of reliability, you could also argue that there is an advantage to a lower-level groupset, because you're riding tried-and-tested technology rather than the "leading edge" technology used by the higher level groupsets (Dura Ace, Ultegra).
So why do shops sell the top-end groupsets? I think there is a big psychological aspect to this, and this is my second point.
On the one hand they are serving the interests of the top-end riders. But on the other hand, they are also catering for a huge bunch of riders (and I include myself in this category) who ride these groupsets simply because they can afford to do so.
As @mattnz says in his comment, it is all about vanity. It's exactly the same reason a guy will drive a Ferrari when he could get get from A to B just as quickly (assuming speed limits are observed) in a Ford.
Last, on a practical note, its worth saying that groupsets cost a lot less when bought as part of a bike, than they do when bought standalone. So if you're the type of rider who will say "if only I had a better bike I could smash my pb", you're better off by going for the superior groupset straight away - it will be cheaper in the long run than getting the lesser groupset and upgrading parts piecemeal.
However, upgrading piecemeal is possible (albeit as I say at a premium). Just note that the current range of Ultegra (6800) is 11-speed, whereas the current 105 (5700) is 10-speed. So to upgrade a 105 part to an Ultegra part, you'd need to look at older (10-speed) Ultegra ranges (6600 or 6700) or even 10-speed Dura Ace (7800). But certainly 6700 is still widely available.
105 is a good choice for most recreational cyclists. At the 105 group level and above the primary differences are not in function, but marginal weight reduction and increased durability.
Some levels of group set give higher level of customisation too, like shorter or longer crank length for example or shifter adjustability. Electric group sets also give some functionality too, i.e. not having to adjust/tune derailleurs and guaranteed perfect shift every time.
Other than functionality there's weight and bling, which could be socially important.
I prefer a balance approach, if you're buying a top level bike, then get Dura Ace to match. If it's a second level bike, then get Ultegra, and so on. Otherwise you will risk someone (like me ;) asking why you put tiagra on a nice bike, you're not doing justice and so on.
You should try a few out to assess their shifting performance. Even better if you can do a blind test. Also consider researching the cost of new parts (eg: chains) and the ease of maintenance.
In my opinion weight is overrated and you can spend a lot of money to make a groupset 100g lighter.