Ellery & Lydia are in the 1850 Federal Census at New Bedford, p. 273, with two sons. In 1860, p. 832, their household includes son Charles, 15, and Hannah S. Hathaway, 31, a dressmaker. She may be Lydia's sister.
In the 1870 Federal Census, Ellery, Lydia, Charles, & Clara are living in Ward 5, Providence, RI. Perhaps Clara is Charles's wife (?).
Ellery's official death entry at Providence City Hall says he was born in Tiverton; died at 37 Claverick St., age 65y 5m 17d; gives the cause of death as disease of the spine; and identifies his parents as Thomas & Alice.