Is it ok to pair 11 speed 11-36T cassette with 50-34T crankset? And is Shimano 105 RD compatible with this setup? I'm planning this setup for my Japanese vintage touring bike.
The Shimano 105 RD in its latest 7000 version has an 11-34 tooth capacity in the long cage version. So theoretically at least, Shimano is rather conservative concerning those limits, the answer is no, you can't. With some tweaking of the b-screw it might be possible but you will have to take the blame if it doesn't work.
This said, the 34 to 34 pairing gives you a 1 to 1 ratio that allows for steep climbs, and is rather demanding in pedalling cadence to keep up enough speed for balance. Example: riding at 6km/h in the 34-34 gear requires a 74 RPM cadence.
Technical details on the the 105-group to be found here: