The estate is located on a 12-metre (39 ft) high hill, the "Heiligenberg" (Holy Mountain), which dates from the last ice age, about 10,000 years ago. On this hill, then known as Hohorst, bishop Ansfridus of Utrecht founded a small abbey around the year 1000. The abbey was moved to the city of Utrecht around 1030, as St. Paul's Abbey, but the Heiligenberg remained in their possession until 1592.
In 1637, the house became an estate. It was torn down in 1843, after which the currently existing building was erected.