Fairhaven was first settled in 1659 as “Cushnea”, the easternmost part of the town of Dartmouth. It was founded on land purchased by English settlers at the Plymouth Colony from the Wampanoag sachem Massasoit, and his son, Wamsutta.

According to the United States Census Bureau, the town has a total area of 14.1 square miles (37 km²), of which 12.4 square miles (32 km²) is land and 1.7 square miles (4.4 km²), or 12.06%, is water. It is bordered by the river and New Bedford to the west, Acushnet to the north, Mattapoisett to the east and Buzzards Bay to the south. The town line with Mattapoisett lies along the Bristol and Plymouth county lines. The town is the southeastern corner of Bristol County, and contains the easternmost point of the county, on West Island. Fairhaven is approximately 54 miles (87 km) south of Boston, 21 miles (34 km) by land west of Cape Cod, and 32 miles (51 km) southeast of Providence, Rhode Island.

The town is located on Buzzards Bay, on the eastern bank of the Acushnet River at its mouth. The lands of the town jut out into the bay via Sconticut Neck and West Island, along with several other small islands. Most of the town’s water area consists of its harbors, bays and coves, along with a portion of the Acushnet‘s waters, and Nasketucket and Scipping Creeks. The town lies along coastal plain, and has some swampland along the Nasketucket and around Boy’s and Girl’s Creeks, north of Priest’s Cove. Fairhaven’s localities include Fairhaven Center, North Fairhaven, East Fairhaven, Oxford, Poverty Point, Nasketucket, Sconticut Neck, and Winsegansett Heights. Most of the town’s population lies either in the west side of town, along Sconticut Neck or in the village of East Fairhaven, with the northeast quarter of the town’s land sparsely populated.

The town has two large public parks, Livesey Park and Cushman Park, as well as a number of smaller ones. Cushman Park, as well as having tennis courts and ballfields and a bandstand, is the location of Fairhaven High School‘s running track. The town has several commercial wharves, a yacht club, and several marinas for recreational craft. There are several small bathing beaches, the largest being the Fort Phoenix State Reservation, a south-facing beach to the east of the fort and the New Bedford Harbor Hurricane Barrier. There is also a bike path which travels along a long-unused railroad right-of-way, just to the south of Route 6.

• John Cook Bennett (1804–1867), physician and a ranking and influential (but short-lived and controversial) leader in the Latter Day Saint movement, who acted as second in command to Joseph Smith for a brief period in the early 1840s
• Emma Borden (1851–1927), older sister of Lizzie Borden, was on an extended visit to friends in Fairhaven on August 4, 1892, when she received a telegram informing her of the murder of her father and stepmother
• William Bradford (1823–1892), marine painter and photographer
• The 1st Baron Fairhaven (1896–1966), multi-millionaire businessman, horse breeder, and art collector who mainly lived in Great Britain; born in Fairhaven but educated in Great Britain, he restored and expanded Anglesey Abbey, his country seat, in Cambridgeshire. Lord Fairhaven was the grandson of Henry Huttleston Rogers
• John Cooke (c. 1607–1695), a Mayflower passenger, Cooke was one of the town’s first European settlers, a Baptist minister, a prominent land owner, and a Representative to Plymouth Court

• Paul Delano (1775–1842), a sea captain, moved to Chile in 1819 where he became an important part of that country’s early Navy
• Warren Delano Jr. (1809–1898),a native of the town, Delano became a prominent trader with Russell & Company, smuggling opium in China. He was the maternal grandfather of US President Franklin Delano Roosevelt and created Fairhaven’s beautiful Riverside Cemetery, where many Delano family members are buried
• Mark Dion (born 1961) noted conceptual artist and sculptor, known for his installations using found human-made and natural objects
• Carl Etelman (1900–1963), football back and coach
• William H. Hand Jr. (1875–1946), one of the most prolific yacht designers of the twentieth century, and whose office was in Fairhaven
• William Le Baron Jenney (1832–1907), architect and engineer who became known as the “Father of the American Skyscraper”; Fairhaven native
• Herman Melville (1819–1891), author of the classic novel Moby-Dick, twenty-one year old Melville stayed briefly in a rooming house in Fairhaven and on January 3, 1841, set sail from here in the whaleship Acushnet.^[14]
• “John” Manjiro Nakahama (1827–1898), the first Japanese person to live in America
• Albert Pike (1809–1891), attorney, soldier, writer, and prominent Freemason. Pike is the only Confederate military officer or figure to be honored with an outdoor statue in Washington, D.C. (in Judiciary Square). A Massachusetts native, he taught school in Fairhaven as a young man
• Christopher Reeve (1952–2004), of Superman fame, the summer resident kept a sailboat, the 40-foot (12 m) sloop-rigged Chandelle, at a Fairhaven shipyard and sometimes flew into New Bedford Regional Airport to pick it up or to stay in town during a stopover en route to Martha’s Vineyard
• Franklin D. Roosevelt (1882–1945), 32nd President of the United States; summer resident
• Gil Santos (1938–2018), longtime radio play-by-play announcer for the New England Patriots of the National Football League and morning sports reporter for WBZ radio in Boston; Fairhaven native
• Frances Ford Seymour (1908–1950), wife of actor Henry Fonda and mother of actress Jane Fonda and actor Peter Fonda; lived in Fairhaven for several years with family members and attended Fairhaven High School
• Joshua Slocum (1844–1909), the first man to sail alone around the world, and his ship, the Spray. The Spray originally belonged to Captain Eben Pierce of Fairhaven, a whaling captain, who gave the derelict boat, slowly deteriorating in a ship cradle in a meadow on Fairhaven’s Poverty Point, to his friend, Captain Slocum. Slocum spent thirteen months in Fairhaven while working on the Spray, making her fit for open-ocean sailing. Fairhaven oak formed much of the boat’s refitted structure. The Spray and her one-man crew returned after nearly three and a half-year to the very cedar spile that was used for her launch. Today, the student newspaper at Fairhaven High School is called “The Spray”
• Noah Stoddard (1755–1850), a privateer captain in the American Revolution
• Theodore Thomas (1835–1905), American’s first renowned orchestra leader and founder of the Chicago Symphony Orchestra, had a Fairhaven summer home surrounded by spacious gardens from 1887 until shortly before his death in 1905
• Mary Ann Tripp (1810–1906), the first American woman to sail around the world and the first American woman to visit China while sailing with her husband, a merchant ship captain
• William H. Whitfield (1804–1886), the sea captain who rescued Manjiro Nakahama and with whom Manjiro lived during his time in Fairhaven