James Hardin Selsor was born on September 4, 1842, in Indiana to Andrew and Nancy Selsor. His father was a farmer who owned $1,500 of real estate and $600 of personal property. The family lived in Brown, Indiana, until the 1850s, when they moved to Medicine, Missouri. By 1860, he was working as a farm laborer.
He enlisted in the Union army on September 1, 1861, and he mustered in as a private in Company C of the 23rd Missouri Infantry later that day. The regiment took part in the Battle of Shiloh, the Atlanta campaign, the March to the Sea, and the Carolinas campaign.
He returned to Missouri after the war, and he married Emma Leonard around 1865. They had at least three children: Frank, born around 1865; Alphonse, born around 1867; and Leonard, born around 1876. They moved to Hamilton, Iowa, in the late 1860s, and Selsor worked as a farmer. By 1870, he owned $1,500 of real estate and $400 of personal property. He applied for a federal pension in April 1889 and eventually secured one. The family moved to Richland, Missouri, in the 1870s. He remained in Richland for the rest of his life, and he died of heart disease there on February 9, 1915.