I am making an assumption that the Joseph and family in the 1860 and 1880 Federal Censuses at Ashford, NY is this individual and his family. Also in the same town are his brothers Copeland and Bela Remington and sister Patty Stevens. (There is the possibility that it may be another Joseph, since it is such a common name.)
In 1860 Joseph is 27, Laretta is 19, and their daughter Nellie is 4 months.
In 1880 they are living with married son Arthur and are identified as his parents; however, in the censuses of 1860 and 1870, Arthur is living with Bela & Silphina Remington and appears to be their son.