James Goforth Willoughby was born on May 7, 1817, near Dover, Delaware, to William Willoughby and Hettie Goforth. The family moved to Collinsville, Illinois, around 1835, and he married Mary Janet Gates on July 30, 1840. They had at least four children: McDonough, born on May 22, 1841; Celestia, born around 1843; James, born around 1850; and Adelaide, born around 1857.
They lived in St. Clair County, Illinois, and Willoughby worked as a farmer. By 1850, he owned $1,800 of real estate. They eventually returned to Collinsville, and by 1870, he owned $10,000 of real estate and $500 of personal property. According to a local writer, he was “active in all the work of the [Methodist Episcopal] church,” and he was “an earnest advocate of all temperance movements.” His wife died in 1873, and he married Mary Thompson around 1879. The 1900 federal census listed him as a “Capitalist.” He died on October 31, 1905.