In the 1850 Federal Census, Elias & Lydia are living at Topsham, Orange Co., VT. In the household are their two children, John & Oliver (I don't know whether there were any older children previously a part of this family group), and Elias's mother, Polly Remington, age 78.
In 1860 Elias & Lydia are still at Topsham. They are living next to his mother Polly, 88, and his sister Mary, 51.
In 1870, Elias is age 70 and Lydia is age 67. They are living at Orange, Orange Co., VT.