Rufus Whitney was born around 1796 in Pennsylvania to Tarball and Lois Whitney. By 1820, he was living in Allegany County, New York. He married Laura M. Pickett around 1839, and they had at least two children: Albert, born on September 17, 1840; and Edwin, born around 1842. By 1850, he was working as a farmer in Almond, New York, and he owned $3,000 of real estate. A decade later, he owned $6,500 of real estate. He died around 1863.