It's probably for the best that optional fights are harder. When I ended up in the pit to face Ifrit, all I could think was REALLY? Fight Ifrit or get stuck here?
Mostly because I was laughably undergeared, not having done a non-story quest in 5 levels or so. Had I been a tank or DPS, I still would have felt bad.
But at that difficulty level, it's not something to stress about. The Sahagin boss guy from the pirate dungeon was harder than Ifrit, for the most part. Though phase 2 of Ifrit was super stressful on my second party through it (DCed after we won on first), since people weren't avoiding AoE and they were already at half life from the unavoidable AoE.
Though it also made me feel badass as a healer.