Amlay House was built in 1858 and belonged to the Amlay family of Simon’s Town until they were forcibly removed on 1 September 1967 under the Group Areas Act of the Nationalist Government. The family were removed in 1975 and were the first former residents to return to Simon’s Town in 1995.