The fence is what marks the outer limit of the outfield portion of the field of play. It runs from the left to the right foul pole. A ball hit over the fence on the fly is a home run; one that passes over the fence on a bounce is an automatic double.
The fence is generally indeed a fence, either made of wood, brick or other such material, and usually although not always covered with protective padding to prevent injuries if an outfielder runs into it. There is usually a warning track laid out a few feet in front of the fence so that outfielders can tell they are nearing the fence. The fence often runs in a semi-circular arc, with its farthest point from home plate in straight center field. There are however hundreds of variations on the shape and size of outfield fences, with everything from pre-existing buildings, hills, trees and waterways having marked the outer limit of the outfield in various ballparks. It is common to find fences made of a series of straight and curving lines joining at various angles, and often of different heights. These quirks are sometimes the result of the physical limits of the grounds on which the ballpark is built (the Green Monster left field fence in Boston's Fenway Park is a famous case) or are simply a design feature chosen to give a park additional character. In many cases when the height or composition of the fence is not uniform, there is a painted yellow line at the top of what constitutes the fence to help the umpires to determine whether a ball has passed over the fence or not.
In theory, there does not need to be a fence at all, as the foul lines are considered to continue infinitely; that is still the case in many Little League or amateur parks where there is no need to accommodate large numbers of spectators, or where the size of the grounds are not limited by man-made obstacles and the possibilty of a ball being hit more than a couple hundred feet is very low.
The official rules only specify a minimum distance from home plate for the fence: 320 feet along the foul lines and 400 feet to center field (rule 1.04), although even that is couched in terms that make derogations possible, especially for pre-existing ballparks. No maximum distance, or minimum or maximum height are set for the fence.
In the earliest rules (the 1845 Knickerbocker Rules), a ball hit over the fence was considered a foul ball and out of play; this interpretation was rapidly discarded however, to be replaced with the current one that there is no outer limit to fair territory.