In the 1820 Federal Census, the three brothers -- Thomas, Silas, & Alva -- are listed at Henrietta, Monroe Co., NY. In 1850 and 1860, Thomas & Huldah are still living at Henrietta, Monroe Co., NY. In 1860 he is listed as Thomas N. Remington, so possibly his middle name was Nelson.
In 1810, he may be at Rupert, Bennington Co., VT, as is Silas. If so, it appears that there are several children in the family.