In 1850 and 1860, Philander and family are at Brutus, Cauyga Co., NY.
In 1860, Susanna Edmunds, age 77 MA, is with them. I am making an assumption she is Philander's mother-in-law. See notes for his wife Sylvia.
In 1870, the household is at Brutus as follows: Phylonder, Sylvia, Joseph, and Eva M. Remington: Anna Lenard 13 domestic.
In the 1874 NY State Census, Filander and Sylvia are still at Brutus; with them is John Remington 16, son. However, I think he is actually their nephew -- unless they have adopted a treenage boy since 1870. I haven't been able to locate nephew John's parents in this census, so perhaps he is living with Philander and Sylvia.