The area was granted on March 27, 1736 by the Massachusetts General Court to a group from Boston. In 1737, the township was laid out and roads cleared, with the first settlers arriving in the spring of 1738. But during the ongoing French and Indian War, the settlement was attacked in the spring of 1745 by Indians and resettled again in 1751. The town had been without a name until about 1756, when it began to be called New Boston. On June 19, 1778, New Boston Plantation was incorporated as Gray after Thomas Gray, a proprietor.
Gray has a number of historic landmarks and buildings within its borders. Pennell Institute is a still standing school building that was built in 1876. The school was considered one of the finest free high schools of its kind in all of New England and is a cornerstone of Gray. In 2009-2010, the Pennell Institute was renovated to house Gray's municipal departments under one roof. This newly renovated Town Hall, aptly renamed the "Henry Pennell Municipal Complex" opened in late summer 2010.
Along Collyer Brook, Samuel Mayall established in 1791 the first successful water-powered woolen mill in North America. Brick ruins and a thick stone wall are all that remains at the site today.