In the 1860 Federal Census, George is age 10 and living with his grandparents in Meriden, New Haven Co., CT. In 1880 he appears to be the George Remington living in Meriden next to Henry E. Remington, most likely his uncle. The census data says his father was born in RI and his mother in CT.
The 1860 Federal Census shows George living with both Hiram Remington and Oliver & Vincy. So, either there are two Georges about the same age, or -- the same George was counted twice in the census.