This guide was written by Jasica/Ayasilkrose, Warrior Officer of Spaceballs The Guild on Draenor, and 70 Druid on Hyjal. It is reprinted here with the author's permission.
I think the easiest way to look at this is to think of magic resistance as armor for spells (although as we'll see in a moment that's only a good analogy in the long run). As with physical attacks, when you cast a spell on an enemy the game builds a combat table and rolls a virtual die to figure out what happens. The spell combat table is a lot simpler than the physical combat table, and includes the following possible results:
The game's scrolling combat text lists a "miss" as "resist," with the word "resist" in white. I'm going to call it a miss because this kind of resist is not (with one exception, see below) dependent on the target's magic resistance. It is strictly a function of the disparity between your level and your target's level, and any +spell hit gear you have.
I don't know what the formula is for spell miss chance, or even if there is a formula. What is known about spell hit is the following: Your base chance to miss against a target of your level is 4%. Your miss chance is capped at 1%. If the target is one level above you, your miss chance is 5%. If 2 levels above you, 6%. At +3 levels, your miss chance is 17% vs. a mob and 13% vs. a player. At +4 levels, 72% vs. mob and 80% vs. player. At +5, 61% vs. mob and 73% vs. player.
A critical hit normally deals 150% damage (as opposed to 200% for physical crits). If you have a talent that increases spell crit damage by 100% (e.g., Ruin), that doubles the 50% bonus, so your spell crits do 200% damage.
Note that this means, most of the time, spell hit increases your dps more than spell crit. 1% worth of spell hit gives you +1% dps, but 1% worth of spell crit gives you +0.5% dps - you gain a "deal 150% damage" slot and lose a "deal 100% damage" slot.
The unfortunate thing is that nobody seems to know exactly. We know what happens, generally, but not the precise math. We do, fortunately, know the long-run average-over-time math.
First we need to define terms. Spells are divided into "regular" and "binary" spells. I don't know if there's a comprehensive list anywhere, but a binary spell is one that has some non-damaging effect in addition to damage, usually a snare. Examples include frost bolt, blast wave, and frost shock. Regular spells are spells that only have one effect pre-talents. Examples include fireball and pyroblast (yes they have a DoT, but it's small enough that it doesn't count as binary) and shadowbolt (even improved shadowbolt, though Imp. SB has two separate effects).
Second, spell penetration. Spell penetration is negative resistance, plain and simple. If the target has 100 resistance and you have 20 spell penetration, his resistance is 80 against your spells. When I say "resistance" in a formula below I mean "effective resistance:" (actual resistance - spell penetration).
So, here's what happens precisely.
For normal spells, there are two rolls. The first is the miss/hit/crit roll. The second, and unrelated roll, is how much of that damage is absorbed by the target's magic resistance. It seems that there are five possible results from the magic resistance roll: take no damage, take 25% damage, take 50% damage, take 75% damage, and take full damage. The game rolls on a table based on the caster's level vs. the target's resistance.
The question you want to know at this point is, for a given resistance vs. a given level of caster, what is the likelihood of rolling 0%, 25%, 50%, 75%, or 100%? The answer is: I don't know. As far as I know nobody knows. We have this page from Blizzard, but not the precise math.
What we do know is what happens on average over time. The average amount of damage resisted is equal to (resistance / caster level x 5) (0.75). The average resistance can never be more than 75%. So if you have 315 fire resistance, vs. a level 63 mob, you will resist (315 / (63 x 5 = 315)) (.75) = 75% of the damage. Unlike damage reduction vs. physical attacks, this is an average. If you have 75% DR from armor you will always take only 25% of the incoming damage (whether that's 25% of a normal hit or 25% of a crit). If you have 75% DR from resistance you might take full damage on one hit, no damage on another, but over time you will average out to taking 25% damage.
That's for normal spells.
For binary spells our information is better. Binary spells only have the miss/hit/crit roll (presumably the reason is because Blizzard considers it silly to take only 50% of a frostbolt's snare). For a binary spell, I think you build the combat table like this (see caveat in next post):
So suppose you're level 70, fighting a level 70 mob with 200 frost resistance. You have +6% to spell hit total and a 12% spell crit chance. You cast a frostbolt. Let's build the combat table:
Our final combat table looks like this:
49.59% regular hit
So you'll notice I said "I think" when I built that combat table for the binary spell above. Here's why I said "I think." What it comes down to is two questions:
1. Do spell crits add the crit bonus after mitigation from resistance, or before?
2. For binary spells, does mitigation from resistance affect crit chance or not?
I'm pretty sure that the answers are "before" and "yes," respectively. Here's why.
For physical crits, the crit bonus is added before mitigation. i.e., if you crit for 200%, the formula is (damage x 2 x mitigation), not (damage x mitigation + damage). So it makes sense that spells would work the same way. My limited tests from my other guild's parsers suggest (but I'm not quite ready to say demonstrate) that this is how spell crits work.
If spell crits do work that way, and binary crits aren't affected by mitigation at all, then binary spells actually end up doing more damage over time against targets with resistance, unless spell crits behave differently from physical crits (I won't post the math here unless people ask but you can probably do it yourself).
Honestly, why wouldn't resistance affect crits for one class of spells, and affect crits for another?
Now of course it is possible that either a) binary spells just are better than nonbinary spells on average, all other things being equal, or b) the two classes of spells are equal, but they're equal because resistance doesn't affect crits for either. But my suspicion is that the way I built it in the original post is correct.
If that's true, and I'm pretty sure it is, then the way you treat binary spells and nonbinary spells is actually exactly the same. If your target has resistance, a nonbinary spell will hit more often but a binary spell will hit harder (maybe that points you in one way or another for pvp; I don't know your preferences), but over time those differences average out to the same place.
No, no, no. Remember that +spell hit, both from talents and from gear, is built into a binary spell's combat table before the hit chance gets slashed by resistance. If you're worried about your target's resistance the only way to mitigate it is spell penetration.
How much spell penetration do you need? Depends on how much resistance your target has. I don't think we have a standard assumption just yet, the way we have a standard assumption of 5% crit/dodge/parry/block chance for mobs, but the consensus seems to be leaning towards your average mob having something like 20-30 resistance, which means 20-30 spell penetration. But that's really not much better than a WAG. And of course in pvp resistances will be all over the board.
1. Unlike physical attacks, +crit and + hit are usually not equal dps in the long run. For spells, in general, +hit is the bigger dps increase. If your class has a talent that raises the critical strike damage bonus by 100%, spell crit and spell hit become equal dps in the long run.
2. How much +spell hit you want depends on how you want to gear. If you want to gear for maximum dps vs. trash or outdoor mobs, you're probably good with +3% hit (maximum hit vs. same-level mobs) and throw the rest into spell damage and crits. If you want to gear for fighting bosses, your best bet is probably to pile on every scrap of +hit you can, even if your crits suffer by comparison. A level 73 (or level ??) boss takes +16% hit to max out.
3. How much +spell crit you want depends mostly on your spells. What you want to ask yourself is, "Besides the fact that crits make me feel good, why do I want spell crits?" An arcane mage's answer, "Because the numbers are bigger," is a bad answer, because the numbers aren't big enough. Remember, before you prefer +crit to +hit on a pure numbers basis your crits have to do more than 200% damage. A frost mage can do that (at least vs. frozen targets). An elemental shaman can't. A destruction warlock has a different reason to prefer crits to +hit (the imp. shadow bolt proc), and might want to pile on +crit even before they reach the +hit cap.
4. When you're dealing with resistances, binary spells hit harder but less often, and nonbinary spells hit weaker but more often. Nevertheless, one is  not better than the other in the long run. Despite the different ways binary and nonbinary spells react to resistance, the answer is the same: spell penetration.
So, there you are: combat tables for spells.