Rufus L. Colby was born around 1835 in New Hampshire to David and Sally Colby. His father was a farmer who owned $1,200 of real estate and $707 of personal property in 1860. Colby grew up and attended school in Warren, New Hampshire, and by 1850, he was working as a laborer on his father’s farm. By 1860, he was working as a miner. He married Louisa A. French on January 25, 1862.
Colby enlisted in the Union army on August 22, 1862, and mustered in as a private in Company G of the 12th New Hampshire Infantry on September 9. According to his service records, he was 5 feet, 7 inches tall, with dark brown hair and blue eyes. The regiment took part in the Battles of Fredericksburg, and he died of disease in Falmouth, Virginia, on February 7, 1863.