James M. Gray was born around 1814 in Maine to Paul Gray and Hannah Marr. He married Emma Swinerton in Sudbury, Massachusetts, on April 22, 1842, and they had at least six children: James, born around 1843; Albert, born around 1845; Harriet, born around 1848; Marshall, born around 1851; Hiram, born around 1865; and Martha, born around 1868. The family lived in Standish, Maine, and Gray worked as a farmer. In 1850, he owned 105 acres of land worth $1,000. That year, the farm produced 7 bushels of rye, 100 bushels of corn, and 44 bushels of oats. A decade later, he owned $1,500 of real estate and $500 of personal property. Gray died in Standish on May 4, 1868.