I had the same question a few years ago. The answer is I think subjective but I came to the conclusion that the same thing goes with clothing as with components. I reached that conclusion by the "suck it and see" approach.
I basically started off along the thought process that winter clothing was more important than summer clothing. By that I mean that you can get away with riding a $20 jersey from China in the middle of summer, and the benefits of a $150 Assos jersey would be less obvious. That's probably not the case, say, with a thermal vest.
On that basis I decided to try some of the more renowned brands with smaller/cheaper winter clothes. For example, gloves and socks. Sure, a pair of Assos gloves will set you back more than a pair of gloves from China, but in absolute terms they're still only a few tens of dollars. So still pretty much affordable, and in particular not $$$s down the drain if they turned out to be no better.
I was very impressed with the quality of these accessories, so much so that since then I have gradually built up a collection of more substantial clothing such as tights and jackets. This stuff is so expensive that it was, like, one garment per year at first. But in every case I have been impressed with the quality. The Assos 851 tights and jackets, for example, are absolutely superb as outer winter layers. Again, I started off with winter clothing.
Back to my initial premise: having been impressed with winter clothing, last year I bought a couple of pieces of top-end summer clothing. Again I was impressed. In particular the chamois in the bibshorts made a lot of difference to the kit I had been wearing, in terms of comfort in the saddle. For an hour or two the difference is acceptable, but over 8-9 hours you would notice a difference I'm sure.
So that's basically how I came to my conclusion. But its also worth saying that you don't necessarily need to buy Assos to get quality goods. Personally I rate Etxeondo as every bit as good, if not better (in terms of quality) as Assos, but cheaper (unfortunately more difficult to get hold of). And you mention Castelli, I have one of their kits which is very comfortable. Also Agu, Santini, Nalini, Moa etc. (Conversely I wouldn't rate replica team kit unless I knew who the manufacturer was.) But a little research will yield results.
I have deliberately ignored the value for money aspect here since it is so subjective. Is a $150 jersey 10 times more comfortable than a $15 jersey? Who's to say?
Lastly, longevity is also an issue here. A mate of mine has just replaced a 10-year-old Assos jacket, and even then only because he tore it. If this stuff is going to last that length of time, it doesn't seem quite so expensive an initial outlay.