About a year ago I swapped the v-brakes with a Tektro Auriga kit. I immediately made a mistake by not properly bedding in rotors and pads, doing "powerful" brakes, sudden stopping of the bike and so on.. (but always on a flat road!). So I immediately felt noticeable vibrations with moderate to strong braking. I'm not talking about annoying noises but rather about vibrations that make the handlebar and the frame vibrate (at the rear), as you can see in this video (enable the audio). Cleaning the rotor many times, properly bedding in, and trying a semi metallic pad, did not solve the problem. Only with a Shimano RT56 rotor I have no vibration, but less powerfull braking. So the question is.. could i have ruined the rotor after a few braking? Or maybe is just a sort of incompatibility between these rotors and my wheels (Mavic Crossride 26 both front and left)? I don't know whether to buy a new Tektro tr-8 rotor or try something else. PS: the Tektro rotor is not bent.
What QR Skewer do you use?
I had the same problem on my QR disc brake bike. After upgrading to BB7 with a good, brand new rotor and good bedding in, the issue got better but did not go away completely. The scariest part was that the QR would just untighten with brake use, so I had to control regularly.
After that I upgraded the stock QR for a old Shimano internal cam one and the problem was completely gone. The issue was not only a weaker cam mechanism, but also that the stock QR had a plastic part that deformed under braking. The QR I now use is all metal.
Take a look at this article for more details: https://handsonbike.blogspot.com/2013/08/difference-between-good-and-bad-qr.html
That kind of vibration can be difficult to eliminate without replacing the rotor.
The window pattern on a given rotor can contribute to it, as can the uneven transfer layer that can develop from not bedding in the rotor properly to start with.
Some improvement can be had by sanding aggressively, then cleaning and re-bedding the rotor. Sanding it off the hub is better because otherwise you'll bias the work on the easy side.
Usually I replace rotors outright to solve this problem because it's more of a guaranteed fix.
Your question is not quite clear, but there is no braking power difference between rotor brands. Differences are solely in weight, design, durability, heat dissipation on long descents etc.
So if your RT56 rotor is not causing vibration, but it is less powerful, then it means that the other rotor is a different size. Otherwise the braking performance would be the same. Rotors of a different size will require an adapter also to fit to the same frame.
It is quite possible that a bigger rotor causes vibration where a smaller one does not, since the braking force is directly proportional to the diameter of the rotor.
I'd expect 160mm to be sufficient rotor size (you don't mention) for most purposes. Remember that most force comes from the front, so for XC MTB some people use 180mm front, 160mm rear. You could experience with using your RT56 rotors on the rear and the Tektro (?) on the front, or vice versa (depending on size).
Also your fork will definitely have a rated rotor size, which relates to the braking force the fork is designed for. If you are putting too much in, then it might well vibrate. You mention 26" wheels, which suggests an older bike - older forks were often 25.4mm, whereas today 32mm is standard. I'd review the rotor size specs for your fork and the rotor size originally fitted to the bike.