Mixing of Shimano 10, 11 and 12 speed derailleurs and shifters is a poor choice because of the incompatability between speed classes. Each of the shifters from these speeds have a differing cable pull from the others--per shift, they don't pull the same length of cable. Differing by only a few tenths of a mm, it still effects the smoothness and accuracy of the system. This then is magnified when a different speed class of rear derailleur is used because there, each of those speed classes of rear derailleur have differing actuation ratios. Actuation ratio is the amount the rear derailleur moves laterally (to move the chain to the next cog) based on the cable pull input of the shifter. While one MAY be able to get a mixed system to work "mostly" or "well enough" through much of the gearing range, the smoothness, accuracy and dependability of the system is lost. It's for these reasons one spends the extra money for XT and XTR components. It's akin to buying a Ferrari and purposely fouling the engine timing--no longer is there the high performance intended from the product.
With that slight rant out of the way--here comes what may seem contradictory to the above preaching, however, to use the Ferrari metaphor, were not messing with engine timing but boosting the already designed performance to get a little more. I read your problem as experiencing poor shift performance in an all 11 speed Shimano system, especially going down the cassette to successively smaller cogs. The 11 speed XT derailleur (RD-M8000) has a max low sprocket size of 47 teeth when used in a 1x system. Shimano specs are notoriously conservative and a little more range (larger low sprocket than defined by Shimano) can be eeked out while still expecting and getting high performance. Normally I would say that jumping out of spec from 47 to 52 teeth is fairly large out-of-spec to expect consistent, smooth shifting but with your use of the goat link and having experienced the XT derailleur's capability (I use it with an 11-42 but the B-adjust screw is basically all the way out. When I've wound it all the way in the gap between the large cog and guide pulley is massive) the excess range your going for is not unreasonable. In fact it does climb the chain onto the large cog.
Poor shifting down the cassette is likely caused by excess friction in the system. The cable tension may be excessive or dirt or other physical impediments to cable movements are present. So start at the beginning and inspect cabling from shifter to derailleur. Disconnect the cable from the derailleur (note whether you've correctly routed the inner cable at the pinch bolt) and taking a section of cabling at a time, manipulate the shifter while holding the inner cable taught with your fingers noting any deviation from perfectly smooth in the cable movement. Worn ferrules or kinks in the outer cable are common points of friction. You've made a lot of changes and if the same cabling was used between shifters, there's lots of opportunity to contaminate or slightly damage the inner cable at change over. Pushing inner cable through the outer moves grease along and abrades the outers' inner liner. If your cable routing is internal and where it exits the frame, if there is a cable clamping mechanism there, make sure that isn't so tight that it's making a dent in the outer cable. Lots of things to look for and excess friction is the number one cause of poor shifting especially down the cassette.
When you reattach the cable, perform the indexing. Start at the second to smallest cog and adjust tension so that putting a little pressure on the shift lever (as if you were to be shifting to one cog larger), so that it's about ½way thru it's throw, adjust the barrell adjuster such that at that ½way point between second and third to smallest cog the chain is making noise because it's rubbing on that third from smallest cog. No noise:loosen barrel adjuster to tighten the cable to get noise. If it shifts before the the lever throw is completed, back off cable tension (rotate adjuster clockwise) until only noise is generated at the lever throws ½way point. Release the lever and the chain should be on the second to smallest cog, the shifter should be at the first detent (rest point after one click). Now attempt the shift to the third from small cog. If it doesn't complete the shift, increase cable tension a little bit until the shift completes. Be conservative, goal is to get just enough tension in the system so we can eliminate the possibilty that poor shifting down the cassette is because of excess tension. Finish indexing by shifting up the cassette adding only enough cable tension to complete a shift if adjustment needs to be done at successive cogs. Very little adjustment needs to be done after the above exercise if the set-up is otherwise correct. When the chain is on the largest cog, check that the distance between the cog and the guide pulley of the derailleur (which should be directly under the largest cog) is 5-6mm. Adjust the B-tension bolt to achieve this distance. Clockwise on the B tension bolt increases cog to pulley distance, counter clockwise narrows the cog to pulley gap.
If you're assured friction isn't an issue and the indexing and B tension are set correctly and still find issues with poor shifting, remove the goat link and start the rear derailleur set up from the beginning, checking limits (shouldn't have to be changed. Little if at all), redo the cable at the pinch bolt, index and set the B tension (which will need to be all the way in to clear a 52 tooth large cog and you may not get the 5-6mm gap but if there's no interference of chainwheel with guide pulley that aspect should be ok).