James Harrison Manly was born around 1840 in Alabama. His father probably died sometime before 1850, and he grew up and attended school in Marengo County, Alabama. He moved to Sumter County, Alabama, in the 1850s. He married Lydia Little on May 29, 1861, and their daughter Lilly was born around 1864.
He enlisted in the Confederate army, and he mustered in as a corporal in Company D of the 42nd Alabama Infantry. The regiment took part in the siege of Corinth and the siege of Vicksburg.
He settled in Jones Bluff, Alabama, after the war, and he earned a living as a farmer. By 1870, he owned $500 of personal property. He moved to Gainesville, Alabama, in the 1870s. He died sometime after 1880.