Can you use a (e.g.) SRAM rear derailleur with a SRAM casette with a (e.g.) Shimano front derailleur with Shimano Chainrings?
The only two ways the front derailer and rear derailer must be compatible are 1) they must be assuming the same chain width (or at least reasonably close), and 2) they must combine to produce a reasonable chainline (more a function of the crankset than the front derailer).
Thus, the main thing to be concerned with is that the replaced derailer be compatible with its associated shifter. If you change the one side shifter to match the derailer, of course, this will not be a problem. I don't know what problems you might encounter attempting to use the old SRAM shifter with the Shimano FD.
Anecdotal accounts say "yes, you can mix within reason".
The manufacturers, however, will generally say a firm "no". On the one hand, they would, wouldn't they? They're in the business of selling kit. With road groupsets, there is Shimano, the main player. Then there is SRAM, who make their components to be largely compatible with Shimano. Lastly, among the main manufacturers, there is Campagnolo, who are compatible with...Campagnolo. For mountain bikes, things are a little different. Shimano still dominate, more arguably,, but SRAM's policy has generally been to buy existing, established brands. At this point I'm guessing somewhat, but I think incompatibility is the general state. Campagnolo aren't even players in the mountain bike marketplace.
With all systems, the width of the outside of thechain really drives everything else. Hopefully you can imagine, as you start putting more and more gears on the back, each cog must sit closer to its neighbour, and so the thinner the chain must become. This follows through to other components also. One of the most critical things is the rear derailleur, which is designed to shift the chain a certain distance every time you change gear. You can imagine that only the derailleur made for that number of gears will shift the chain the correct distance. If you mix this, the distance will not be quite right, although we're talking tenths of mm here between each gear. So whilst the shifting is not perfect, you may judge it to be "good enough" by your standards. In that sense, it's a judgement call.
Lennard Zinn, who has authored a few excellent books on bike maintenance, has also written quite a bit on mixing over on Velonews.
The specific swap you describe may work just fine since nearly all SRAM chainrings and cogsets use the same spacings as Shimano. So, mechanically, the SRAM cogs and Shimano cogs are interchangeable under whatever FD/RD you have.(Campagnolo, not so much!)
This assumes you are using a briter/shifter set that is compatible with your FD+RD combo. (e.g. early D.A. or something else wacky won't work) That's the real potential issue.
There are also concerns matching brake lever pull (range/ratio) if you use brifters too. Those aren't even guaranteed to match across the same MFG line anymore... (e.g. Shimano's ultegra recently had a change to the brake pull setup... they'll still 'work' with mismatched components--old&new ultegra, just not as shimano intended--Brakes are one of the few dedicated safety systems on a bike, so make sure you're extra careful with mixing MFG/Lines there.)
In the end, it is all just simple ratios and lengths. https://en.wikibooks.org/wiki/Bicycles/Maintenance_and_Repair/Gear-changing_Dimensions may help explain some of this more in depth. (as the question is a little vague for a really short answer.. e.g. would need mfg part#'s for chanrings, cogs, and shift-levers--which are not provided)
So, generally, for indexing to work, which I think is your concern... the gears really don't matter as long as the levers match them and the levers have a proper pull for the FD/RD. (e.g. worry more about the lever compatiblity than the RD/FD)
However... After the swap, the bike will shift differently. The big-three all sound slightly different when they change gears, so don't be surprised if the bike feels/sounds a little different after the swap. A (new) chain that matches the rear cluster can help, but YMMV... your old chain may do just fine too!