Ira Colby was born on December 31, 1803, in Henniker, New Hampshire, to Horace Colby and Nancy Fellows. He married Polly Foster in Henniker on April 17, 1827, and they had at least seven children: Zebulon, born around 1828; Ira, born around 1831; Adeline, born around 1833; George, born around 1836; Etta, born around 1839; Helen, born around 1842; and Eliza, born around 1844. They moved to Claremont, New Hampshire, in the late 1820s, and Colby worked as a farmer there. By 1850, he owned $6,000 of real estate. A decade later, he owned $5,500 of real estate and $1,000 of personal property, and by 1870, he owned $9,600 of personal property. He died in Claremont on July 7, 1873.