Ellen M. Madison (maiden name: Glossbrenner) was born around 1835 in Ohio to John Glossbrenner and Elizabeth Austin. Her father probably died before 1850. She grew up and attended school in Blooming Grove, Ohio. She moved to Illinois in the 1850s, and she married James Madison on November 4, 1858. They lived in Wabash, Illinois, and her husband worked as a physician. By 1860, they owned $300 of personal property. A decade later, they owned $1,200 of real estate and $2,500 of personal property. They adopted a daughter named Katie in the 1870s. Her husband died in 1905, and she died in Marshall, Illinois, on March 27, 1909.