Sap is most often utilized in instance, where a mob needs to be removed from the battle to make the fight safer. A group may choose to sap an enemy healer or caster; more often, a high-damage target is sapped to help keep the tank alive, and the healer is killed quickly. (The reasoning is such: the highest rank of Sap only lasts 45 seconds. If any member of the group is still alive when Sap breaks, a healer mob may return to heal that enemy before it is noticed, thus prolonging the fight.)
Sap is also a useful tool in PvP. Removing an enemy player from the fight can give one team a huge edge, particularly in the Arena. It is not uncommon for two teams' rogues to stealth before the fight and spend their time looking for each other, trying to sap one another. In battlegrounds or world PvP, Sap grants less of an edge but is still useful. Note that Sap does not last the tooltip duration in PvP, and is subject to diminishing returns.
To use Sap, one must be in stealth and the target must be out of combat. If the rogue is spotted, both of these conditions will change; therefore it is crucial to know how to sneak up on a target.
One essential yet oft-overlooked tool is Distract. This ability will cause all targets within the area of effect to turn and face the center of the targeting circle. It is possible for Distract to be resisted; if this happens, you must judge whether you can still reach and sap your target safely. It is much less disastrous to wait out the 30-second cooldown than to wait for your group to run back to the instance from the nearest graveyard!
Also, remember that you are much less likely to be spotted when approaching from behind. In fact, unless you outlevel the distance by a fair bit, approaching from the front or too much to one side will cause you to be seen. If you use Distract wisely, it will almost always be possible to reach your target unseen.
Last but not least, do not pick your target's pockets! It may seem common sense, but some rogues will attempt a Pick Pocket while sapping. Pick Pocket can be resisted, and at some point probably will, so don't do it. And if, despite all your precautions, you are seen by the group, a quick Vanish will reset them - just make sure to get out of the way first or you'll be seen again.