In the 1850 Federal census, James & Caroline are living at Alden, Erie Co., NY. They have two sons: George and James. James's sister Sarah Remington, 40, is also living with the family.
In 1860 they are still at Alden, NY, and in 1870 they are Lancster, Erie Co., NY. In 1870 James Remington is the Assistant Marshall who took the census. His sister Sarah is still living with the family, as well as two other boarders.
According to the 1850 census, James was born in MA, but the 1860 says NY. It might have been either; his brother David was born in MA, but sister Sarah was born in NY.
[S012551] Boyd Scott Remington, Remington Lines from John of Newbury 1638, (Ames Iowa, 1960), 7.
[S013166] James N. Granger, Launcelot Granger of Newbury, Mass. and Suffield, Conn.: a genealogical history, (Hartford, CT., unknown pub., 1893.), 142.