The word "instance" itself refers to a dungeon where only one person or group can enter at a time. Of course, there're multiple copies of the same dungeon, but you can only enter one copy, and you'll have to wait awhile until you can go back into a different one. The "instance" idea practically eliminates botting and such. Here's an example, based on my understanding:
Eight people come up to the entrance of an instanced dungeon. Player 1 and 2 (who are partnered up) enter and end up in Copy A. Player 3 tries to go in, too, but he is placed in Copy B, as he isn't in the same group as the first two. Then, Players 4-7 enter as a group, securing Copy C all for themselves. Being all alone, Player 3 invites Player 8 into his copy of the dungeon.