Basically, a guild is a collection of players united under a single collective name. The reasons behind joining a guild vary as widely as the reasons for playing WoW in the first place. The guild name itself will appear underneath a player's name.
Guild organization varies from guild to guild, but typically they each will have members, officers, and a leader. The actual composition and designation of the ranks is left purely up to the whim of the leader.
Guild members form up the bulk of the guild. Within "members" there may also be several ranks denoting different levels of the guild. For example, a guild may have "New Members" (or recruits), "Social Members", and "Raiding Members".
Officers form the leadership of the guild. The actual permissions of officers are set by the Guild Leader. Responsibilities vary by each guild, but typically an officer can expect to handle the common day to day leadership tasks of the guild.
The Guild Leader is responsible for the guild as a whole. Commonly the leader will be responsible for making guild-wide decisions, setting policies, and running the guild as a whole.
The guild window shows basic information about the guild members. By default, it shows the guild stats of online members (the image to the right). Additionally, you can toggle it to show offline members as well as member status (which shows rank, public note, and last time online).
The Guild Message of the Day (or GMotD) is a short (128 characters maximum) message that appears on the guild window. It also appears in a member's chat frame each time they log in. In the example image, you see Cartel's current raiding schedule along with the fact that Felweed is welcome back to be deposited into the guild bank.
The permissions for editing guild information are controlled by the guild leader.
There are a few things to consider when forming a guild. What type of guild is it going to be (and what will be the primary focus)? How should the guild policies be laid out? Who is trustworthy enough to perform the role of officers?
The actual act of forming the guild is rather simple though, simply speak with the Guild Master in a major city to buy a Guild Charter. Find 10 people to sign the charter and then register it back at the Guild Master.
There are as many different types of guilds as there are people who make up guild membership.
A social guild's main goal is simply to exist for social reasons. They typically span the ranges of levels, classes, and member goals.
A leveling guild is much like a social guild, with the exception that there is commonly more emphasis placed on leveling and doing instances and quests as a group.
A raiding guild exists mainly to raid high end content. There are different styles of raid guilds, from the hardcore bleeding-edge progression guilds to the more laid back style of raiding guild. These types typically have higher standards for gear and character progression than either social or leveling guilds.
A PvP guild's primary interest is in the PvP aspect of WoW. They will typically focus on pre-made battlegrounds groups and arena teams.
There are also many guilds that do not fall into the above categories (or are combinations of).
See Guild Leadership for details on leading a guild.