I'm going to go ahead and say that SMN has been my favorite job to level up, especially from Levels 1-10. Those just flew by.
In parties, it's a different story. I worked the Shakhrami Maze from 10-13, and I kept getting invited to mage-only parties that would cast Carbuncle and run behind a corner. I obeyed, obviously, because Summoners have less than, like, 25 HP...
And, of course, there's the Dunes, which I don't think is pleasant for anyone. I didn't have the "Would you be our main healer?" problem until last night, and in that case,
everyone asked me to be the main healer.
People, go read the SMN forum if you don't believe me. We can't be main healers. We don't have Cure II, and if one of you gets nuked, you're dead.
Now, as far as the worst job to level, I'd have to say it's a tie between WHM and THF. THF was good for me until about Lv5, when things just stopped dying fast enough. And I don't think I ever enjoyed WHM. I felt like a sitting duck with whatever I was fighting.