Casting has two big problems.
1. How Magicka pools, magicka costs, and gear advance relative to each other. You start hitting some really annoying points where you are going OOM just way too fast, but downgrading your spells is too big a damage loss. It's rectified once you get stronger cost reduction gear, but it's annoying early on.
2. The fact that you have no way to buff your destruction damage. You are essentially going to plateau at 90 damage per spell (though fire can go higher, if the target is on fire, and the other two schools will damage stamina or magicka). I don't remember how strong doublecasting makes it, though. Powerful, sure. But compared to the damage that weapon-based characters can get, even assuming they aren't combining alchemy/smithing/enchanting to ludicrous ends? It just doesn't compare, imo. I really do think that Fortify Destruction should buff its damage.