James Elmsley was born around 1832 in Scotland. He eventually immigrated to America and settled in New York. He enlisted in the Union army on May 21, 1862, and he mustered in as a corporal in Company B of the 132nd New York Infantry. According to his service records, he was 5 feet, 5 ½ inches tall, with brown hair and gray eyes. In January 1864, he was promoted to 2nd lieutenant of the 35th USCT Infantry. He was promoted to 1st lieutenant in March 1866, and he mustered out on April 25, 1866. He died sometime afterward.