James D. Dameron was born on September 4, 1815, in Virginia. He probably grew up in Norfolk, Virginia, and he moved to New Orleans, Louisiana, around 1836. He married Camilla D. Palfrey, and they had at least six children: Edward, born around 1850; Walker, born around 1856; William, born around 1858; Everett, born around 1860; Frank, born around 1861 and Mary, born around 1867. His wife died on July 18, 1868, and he passed away in New Orleans on June 16, 1869.