Valdemar's Castle was commissioned by King Christian IV (1588–1648) and built between 1639 and 1644 under the plans designed by architect Hans van Steenwinckel d.y. (1587–1639). It was not a fortification, but rather a large manor house. King Christian was well known for his interest in building. Tåsinge, the island where Christian planned to have Valdermar's Slot built, belonged to his mother in law Ellen Marsvin. He intended the new mansion should become the home of his son Valdemar Christian (1622–1656) whose mother was Kirstine Munk. Valdemar Christian never occupied the manor but was killed during a battle in Poland in 1656. In 1678 the naval hero, Admiral Niels Juel (1629–1697) was given title to the castle and the land on Tåsinge after his victory over Sweden in the Battle of Køge. The estate was transferred to him as payment for the Swedish ships captured in the battle.
- Caroline Fleming (born 1975 as Baroness Caroline Elizabeth Ada Iuel-Brockdorff, at Valdemar's Castle) is a Danish noble, entrepreneur, model and TV personality; she was the owner of Valdemar's Castle from 2003 to 2011.
The present owner, Baron Iuel-Brockdorff, who is 11th generation of the Juel family, took over his childhood home from his father in 1971 and lives in the castle with his wife and family. Valdemars Slot has been open to the public since 1974. The mansion is open from May to October and on public holidays. It features a large chapel, a toy museum, the Iuel-Brockdorff family's big game trophy collection and a local maritime museum. As it stands close by the shore, many visitors make the trip from Svendborg using the veteran ferry Helge.