James W. Libby was born on February 28, 1840, in Leeds, Maine, to Pelatiah and Betsey Libby. His father was a farmer who owned $1,800 of real estate and $700 of personal property by 1860. He grew up and attended school in Leeds.
He enlisted in the Union army on August 21, 1861, and he mustered in as a corporal in Company K of the 7th Maine Infantry. The regiment took part in the Peninsula campaign, the Battle of Chancellorsville, the Battle of Gettysburg, and the Overland Campaign. He was wounded in the Battle of Spotsylvania Court House on May 12, 1864. He eventually recovered, and in November 1864, he became a 2nd lieutenant in the 1st Maine Sharpshooters Battalion. Then, in the spring of 1865, he became captain of Company K of the 14th Maine Infantry. He mustered out on August 28, 1865.
He married Octavia Berry on January 21, 1864, and they had at least two children: Ida, born around 1867; and Bertha, born around 1869. They lived in Greene, Maine, and he worked as a farmer. By 1870, he owned $1,000 of real estate and $400 of personal property. His wife probably died in the early 1870s, and he married Rosa Burnham on September 27, 1871. They had at least four children: Edwin, born around 1873; Bessie, born around 1875; William, born around 1879; and Eunice, born around 1883. They moved to Hartford, Maine, in the 1870s, and he worked as a farmer and carpenter.
By 1890, he was suffering from rheumatism. He applied for a federal pension in July 1890 and eventually secured one. He died of typhoid pneumonia in Hartford on June 23, 1912.