I am not sure which is his first name. In the 1860 census he is Rufus, but in the 1870 census he is Edmund. It appears to be the same family group in every other way.
In 1880 he is Rufus E. Remington, and his wife is Ruth C (?). They have three children and are living with Rufus's parents, Allen & Susan Remington, At Ellisburgh, Jefferson Co., NY.
In the Federal Census of 1 June 1900, Rufus E. & Ruth are at Sandy Creek, Oswego Co., NY. They have two daughters, Dora 20 & Kate 7.
In the Federal Census of 1 Jan 1920, Rufus & Ruth Remington are living on North Ridge Road in Sandy Creek, Oswego Co., NY. Their widowed daughter-in-law Leona and family are also in Sandy Creek.
In 1920 he is still at Sandy Creek, a widower age 79, listed as father-in-law living with Ward E. & Kate R. Crandall and their two sons.