Joseph E. Brown was born around 1818 in Ohio. He married Mary Richmond on March 18, 1844, and they had at least three children: Elizabeth, born on August 30, 1845; Allen, born on December 17, 1849; and Joseph, born on October 31, 1859. They lived in Newton, Ohio, and he worked as a wagon maker. By 1850, he owned $600 of real estate. A decade later, his wealth had grown to $1,200 of real estate and $700 of personal property.
He enlisted in the Union army on November 2, 1861, and he mustered in as a private in Company D of the 6th Ohio Cavalry. The regiment took part in the Battle of Fredericksburg and the Battle of Gettysburg. He was promoted to sergeant on December 14, 1861. He died of erysipelas in Portsmouth, Rhode Island on May 19, 1864.