James Wilson was born on November 24, 1806, in Scotland to William and Margaret Wilson. He immigrated to America around 1827 and settled in Hanover, New York, near an uncle’s household. According to an early biographer, he “built a cabin in the midst of dense woods and began to clear the tract of land upon which they had settled.” He married Philena Davison, and they had at least eight children: John, born around 1836; William, born on December 23, 1844; James, born around 1847; Sylvia, born around 1849; Henry, born around 1854; Mary, born around 1857; Prescott, born around 1859; and Walter, born around 1863.
He worked as a farmer in Hanover, and by 1860, he owned $3,700 of real estate and $600 of personal property. A decade later, his wealth had grown to $7,800 of real estate and $2,000 of personal property. He supported the Republican Party. His wife died in 1868, and he married Harriet Flint soon afterward. She probably died in the early 1870s, and he married Matilda Johnson sometime before 1880. He died in 1891.