I am making an assumption that Lydia's middle initial "A" stood for Alice, and that she was known by that name. The Durfee genealogy book shows an Alice Remington, born to John & Hannah about this year, who married Amanezer James Durfee (see book The Descendants of Thomas Durfee, p. 352, by William Field Reed).
In my research at New Bedford I jotted down the date of death for an Alice (Remington) Durfee, 30 May 1878 at Tiverton, age 37 (don't have source noted). This seems to be this individual.
Alice Remington became guardian for Nathan & Harriet Remington after the death of both John & Albert.