Can someone explain to me the difference between the three and where they would apply? Magic Damage Taken, Magic Defense Bonus, and a Resist set.
MDT = A flat % reduction to magic damage taken, Caps at 50%
MDB = Adjust formula on how much damage is taken
Resist = Adds elemental resistance for a single element. The higher the resit the more resists you get and less dammage you take on average.
I would really like to know when I should use which sets ect.
Depends on what gear you have and how serious you are.
Me? I cheat since I have Aegis and all my WHMs have merited Shell 5. That caps MDT right there so MDT- Is not needed and I go with MDB for general use.
MDB has no known cap and depend on on sub job and buffs is quite easy to get into the 50+ range which on most mobs is a good as a resist set.
MDT/MDB is best when you don't know what element you will be against, the element changes (for example JoL), or you can not build a sufficient resist set for (for example: Light/Dark aka: Proto-Ultima).
If you know what element you will be getting hit with then with a resist set look at how much resist you have with your setup. For example with BRD and WHM with merited BAR spells and proper gear for Earth and Fire I am at 320+. I actually ran into issues where I don't take enough damage in my resist sets to have HP to cure hold hate with out massive HP- -> HP+ cure cheat macros. If the mob is higher level then you then you need to aim for ~300 resist with buffs. If you don't have 250+ Then you are probably better off with MDT/MDB.
For MDB vs MDT a lot comes down to what you have for support.
If your WHM has full merits on Shellra V (They should) then the Magic Damage reduction is 27.34% to start. Can you get 23% more in gear? If you can then you can put MDB in the reset of the slots.
For MDB you want to aim for something in the 30+ range to start. My base gear is +38 MDB, +4 MD, +4M.Eva and I'm still missing some of the newer pieces that would raise that since NMs are being stingy. The Shamshir +1 sword is a must for this set. About the single best piece you can get.
If the mob does physical attacks also then you want a shield on which means things like the Iron Ram Lance are out and you'd want to keep AF+1 feet over Damage - feet. That tends to push the MDB over the MDT. If you can stand out of Melee range then MDT starts to move up.
Personally I carry a PDT, MDB, DD, and face tank set on me at all times. I keep resist sets on mules for fights that are planned for specific NMs or pop windows. I also keep a breath damage set for random NMs in storage (most is R/E).