James A. Riddle was born on April 22, 1828, in South Carolina, to William Riddle. His father was a farmer who owned $900 of real estate by 1850. He married Rosa Caldwell, and they had at least five children: John, born around 1850; M., born around 1852; Andrew, born around 1855; Mary, born around 1857; and Emma, born around 1860. They lived in Laurens District, South Carolina, and Riddle worked as a farmer.
He enlisted in the Confederate army on January 2, 1862, and he mustered in as a private in Company E of the 3rd South Carolina Infantry Battalion. The regiment took part in the Second Battle of Manassas, the Battle of Antietam, the Battle of Fredericksburg, the Battle of Chancellorsville, the Battle of Gettysburg, the Battle of Chickamauga, and the Overland Campaign. He was promoted to corporal in March 1862, to 2nd lieutenant in April 1863, and to 1st lieutenant in July 1864. He received a parole at the end of the war on May 2, 1865.
He settled in Youngs, South Carolina, after the war and resumed his work as a farmer. His wife died in 1889, and he married Lou Caldwell around 1890. Their daughter Lou Belle was born around 1894. Riddle served as a deacon in the Durban Creek Baptist Church. He died on March 16, 1905.