James Scarbrough was born on October 18, 1807, in Pennsylvania. He married Elizabeth Kershaw in Philadelphia, Pennsylvania, on October 23, 1836, and they had at least eight children: Harriet, born around 1836; John, born around 1838; Thomas Jefferson, born around 1841; William Henry, born around 1842; Mary, born around 1844; Charity, born around 1846; Amos, born around 1848; and Maria, born around 1851.
They moved to Liberty, Ohio, in the 1840s, and Scarbrough worked as a farmer. By 1850, he owned $1,800 of real estate. A decade later, he owned $4,720 of real estate and $1,910 of personal property, and by 1870, he owned $7,200 of real estate and $1,152 of personal property. He died on October 9, 1896.