DoT is an acronym for "Damage Over Time".
It refers to any spell (such as Shadow Word: Pain or Immolate or ability (such as poisons or Rend that applies periodic damage over a period of time. Most often it refers to the spell variety when someone says "DoT that stuff up". They don't really mean to go rend it, they want the warlock or priest to throw some DoT spells at it.
Note that DoTs will break the application Crowd Control on a mob. Also, DoTs on your character will interrupt bandaging and other channeled abilities.
See also HoT