After the death of his wife Charlotte in 1843 (probably at Albany, NY), Joseph and the children went to Michigan, then to Ohio, then back to Mason County, MI, where they settled. (see posting of 10 March 2000 by Shirley McClain at Rootsweb's Remington message board)
I can't find Joseph in either the 1870 or 1880 Federal Censuses in MI.