Charles Dolphin Brown was born around 1846 in New York to John Brown and Lucinda Morris. His father was a farmer who owned $300 of real estate and $200 of personal property by 1860. He grew up and attended school in Otto, New York, and he married Delilah Babock around 1866. They had at least three children: Willie, born around 1867; Myrtle, born around 1871; and John, born around 1876. By 1870, they were living in Sparta, Michigan, and Brown was working as a farm laborer. They moved to Otto, Pennsylvania, in the early 1870s, and he died sometime after 1880.