Thomas Jefferson Brown was born on October 23, 1848, in Virginia to Madison Brown and Martha Hopkins. His father was a farmer who owned $4,000 of real estate and $1,000 of personal property by 1860. The family lived in Rockingham County, Virginia, until the 1850s, when they moved to Crooked River, Missouri.
He married a woman named Caroline in the 1870s, and they had at least two children: Pocahontas, born around 1880; and Katie, born around 1883. Brown worked as a farmer in Crooked River. His wife died in 1889. He remained in Crooked River for the rest of his life, and he died there of cirrhosis of the liver on April 5, 1929.