Drury J. Brown was born around 1813 in North Carolina. He married a woman named Sarah, and they had at least three children: Edward, born around 1840; Abbie, born around 1842; Ringgold, born around 1846. By the early 1840s, they were living in Hinds County, Mississippi. By 1850, he was serving as a justice of the peace, and he owned $800 of real estate. He was elected sheriff in 1851, and he became a local postmaster in December 1852.
His wife died in the 1850s, and he married a woman named Elizabeth. They moved to Copiah County, Mississippi, in the 1850s, and Brown earned a living as a farmer. By 1860, he owned $23,000 of real estate and $86,000 of personal property. He received a commission as a captain in Company K of the 36th Mississippi Infantry on March 7, 1862. He was promoted to colonel soon afterward.
In the fall of 1865, Freedmen’s Bureau agents arrested Brown on “some charge of violence to a freedman.” The governor lobbied to transfer Brown’s case to a civilian court, and he was ultimately released. Brown died in February 1868.