Pigtails, remember that you may have mods that are making things easier. Combat Tweaks and Advanced Tactics would do that. Plus, Mages are super OP. And don't forget that there's a good chance you are using items from mods with stat levels that would otherwise be unavailable at your current place in the game.
I didn't find the difficulty to really affect how hard mobs were. Rather, they just made me pay attention to my companions so I wasn't constantly friendly-firing them.
TL;DR: It's because the mobs don't get harder.
That's because it doesn't affect mobs much at all. The biggest change is Friendly Fire (Casual = 0, Normal = 50%, the other two = 100%).
Casual adds a flat bonus to defense, attack, and damage. But the other 3 don't.
Healing does 150% on Casual, 100% on normal, then 85% on the other two.
Damage Threshold is another big one. On Nightmare, your damage will be reduced by 16 per attack, where it's only 3 on Casual (7 and 9 for the other two).
Enemies have a 10% lower chance to resist on Casual, and a 5% higher chance on Nightmare (1.5, 2.5).
You get a 10% higher chance to resist on Casual, everything else has no bonus.
You'll loot potions and poultices far more frequently on Casual difficulty than on others. You'll never loot them from random drops on Nightmare (if I'm understanding the wiki page correctly).
Traps do 50% more damage to you on Nightmare (and only do 50% to you on casual).
Finally, and what is probably the biggest change, is that enemies have full AI on the higher two difficulties, meaning they'll use more complex tactics and all of their abilities. But the actual enemies aren't really doing much more damage, they're just organizing more, and you're doing more damage to your own team. End result is you pay more attention to your team overall, to counteract the enemy's increased tactical ability. You also have to monitor healing more closely.
Biggest change to mobs is really in your ability to damage them, since they'll resist 5% more often and take 15 less damage per hit than on casual.