James Scott Hanna was born on February 5, 1819, in South Carolina. His family moved to Alabama when he was a child. He married Martha A. E. Cole around 1845, and they had at least seven children: Eudora, born around 1848; Emma, born around 1850; James Knox Polk, born around 1852; William, born around 1855; Mollie, born around 1863; Mattie, born around 1863; and Heywood, born around 1865. The family moved to Arkansas and then Texas sometime in the late 1840s. In 1854, they settled in Robertson County, Texas, and Hannah established a plantation along the Brazos River. He owned a grocery business and helped extend the Houston and Texas Central Railroad into Robertson County. He died on April 14, 1886.