Thomas Jefferson Frey was born around 1821 in Robertson County, Tennessee, to Peter Frey and Sarah Binkley. His father was a farmer who owned $500 of real estate in 1850. He married Elizabeth Jane Farthing on December 20, 1841, and they had at least nine children: Wesley, born around 1842; Dewitt Clinton, born around 1845; Cornelius, born around 1847; Lysander, born around 1849; William, born around 1851; Sarah Ann, born around 1852; Mary, born around 1855; Richard, born around 1857; and Theodocia, born around 1859. Frey worked as a farmer in Robertson County, and by 1850, he owned $200 of real estate. The family moved to Montgomery County sometime in the 1850s, and by 1860, they owned $3,300 of real estate and $1,150 of personal property. Despite only being in his early 40s, Frey apparently avoided Confederate military service during the Civil War. By 1870, he owned $12,000 of real estate and $2,000 of personal property. He remained in Montgomery County for the rest of his life, and he died there on January 31, 1895.