I have placed Betsey as this particular Joseph's wife, and these children as his children, based on his date of birth and his age in the various censuses; also, the location of their parents' birth in the 1880 census agrees. Therefore, I have made an assumption that this family belongs to this Joseph born 21 Dec 1798.
In the 1850 Federal Census, the family is living at Kingsbury, Washington Co., NY:
Joseph Remington 52 VT farmer
Betsey 43 VT
Gilbert 23 VT
Robert 20 VT
Albert 16 VT
Allen 11 VT
In the 1860 Federal Census, the family is living at Hartford, Washington Co., NY:
Joseph Remington 61 VT farmer
Betsey 53 VT
Robert 30 VT
Albert 26 VT
Allen 21 VT
Ann 22 NY (possibly Albert's wife?)
In the 1870 Federal Census, the family is living at Fort Ann, Washington Co., NY:
Joseph Remington 71 VT farmer
Betsey 63 VT
Gilbert 43 VT
Robert 40 VT
In the 1880 Federal Census, they are living at South Bay, Washington Co., NY:
Joseph Remington 81 VT (MA/VT) farmer
Betsey 73 VT (CT/VT)
Robert 50 VT (VT/VT) (born without use of his limbs)