I would say don't even bother with it.
You need to find 2 stones to complete it.
One drops from a caster that spawns in one of the tents next to the tower, the other from a captain that spawns and walks a bit around the tower, I found him inside.
After you get both you talk to the orb and it will give you an item that when used it will port the people in the tower to the 2nd floor. In there you can face Radavex (47 caster). He drops a crappy robe called Radavex robe.
If you go ahead and try it, good luck!