In the 1860 Federal Census, Charles & Lydia are living in Johnston with their three oldest children, Almy D. 8, William S. 5, & Charles H. 3. They are next door to Charles's grandparents, Charles & Lydia, and right near his parents and siblings.
In the 1870 Federal Census, the family is living in Johnston, RI. Charles and the two oldest children are working in a cotton mill. In 1880 the family is living in Pawtucket, RI.