Albert D. Swift was born around 1840 in Massachusetts to Erastus Swift. His father was a farmer who moved the family to Hartford, New York, around 1846. His mother probably died around 1850, and his father died of bilious pneumonia in Sandisfield, Massachusetts, around November 1859. Swift remained in New York, purchased or inherited a plot of land, and began working as a farmer. He married a woman named Harriet, and they had at least two children: Clarence, born around 1871, and Mabel, born around 1884. He moved to Lee, Massachusetts, sometime in the late 1870s. His first wife probably died in the 1880s, and he married Hattie E. White on February 19, 1889. Their son Howard was born around 1893. Swift remained in Lee for the rest of his life, and he died there on April 17, 1922.