For part of his early childhood Joseph was brought up in a Shaker community at New Lebanon, NY.
According to the book "Remingtons of Utah," Joseph & Sarah moved to western NY state from Basin Harbor, VT, sometime between 1810 and 1818. Around 1830 they were living in Hayfield, Crawford Co., PA; in 1840 they were in Venango, PA. By 1842 they had joined the Mormons in Nauvoo, IL, along with their two youngest children. The family was probably in the wagon train that left Nauvoo for Council Bluffs in 1846. Joseph probably died in Council Bluffs, IA, or in Winter Quarters, NE, across the Mississippi River from Iowa.