Joseph W. Swartz was born around 1851 in Pennsylvania to Andrew and Catherine Swartz. His father was a coach maker who owned $1,500 of real estate and $250 of personal property by 1860. He grew up and attended school in Plumstead, Pennsylvania, and he married Emma Seese on July 22, 1873. They had at least five children: Bertha, born around 1875; Elnora, born around 1877; Rena, born around 1879; Nellie, born around 1881; and Paul, born around 1884. They lived in Plumstead, and Swartz worked as a cabinetmaker. They moved to Trenton, New Jersey, in the late 1800s, and he died sometime after 1900.