James Daniel Waddell was born on December 22, 1832, in Abbeville, South Carolina, to Isaac W. Waddell and Sarah Daniel. His father was a minister. The family moved to Georgia by the 1840s, and his father died there in October 1849. Around 1850, Waddell enrolled at the University of Georgia, where his uncle James was a professor, and he remained there until at least 1851. He eventually settled in Polk County, Georgia, and he married Medora Sparks there on January 14, 1857. By 1860, he was working as a lawyer, and he owned $2,500 of real estate and $18,000, including at least 14 enslaved laborers.
On June 1, 1861, Waddell received a commission as a captain in Company D of the 20th Georgia Infantry. He was promoted to major on June 14, 1862, and eventually received another promotion to colonel. The army assigned the regiment to the Army of Northern Virginia, and it took part in the Seven Days Battles, the Battles of Antietam, Fredericksburg, Chancellorsville, and Gettysburg. His health deteriorated after the Battle of Gettysburg, and Confederate officials named him president of the Slave Claims Board in Richmond, Virginia, in November 1863. He surrendered to Union forces in Augusta, Georgia, on May 23, 1863, and received a parole later that day. Waddell returned home after the war, and he eventually moved to Cobb County, Georgia. Medora died of breast cancer on March 11, 1880, and he followed on December 15, 1881.