Pforzheim was the residence of Margrave Charles II, until he decided in 1563 to move to Durlach. What, if anything, the citizens of Durlach offered him during the negotiations is unknown. Charles decided to extende the medieval Karlsburg Castle into a palace and make it his residence. His successors further extended the castle, until the city was occupied by French troops in 1689. The French burned down the city and the castle. In 1698, Margrave Frederick Magnus returned from exile in Basel and started rebuilding the castle. Frederick had grandiose plans, but no money and the margraviate had been devastated by the war. A dispute erupted with the citizens of Durlach, who refused to support the rebuilding effort. By 1703 two wings had been completed and the margrave had moved in, when the project was suspended indefinitely.
Frederick's son and successor, Margrave Charles III William, decided in 1715 to end the discussion and relocate his residence to a new palace outside Durlach. He founded the city of Karlsruhe, centered on his new palace. Charles and his court moved into the new palace in 1718. His wife, however, chose to remain in Durlach until her death in 1743.
The Karlsburg was later used as an administrative office and even as a barracks. The Pfinzgau Museum has been housed in the castle since 1924. In 1964, a wing was demolished to make room for the Castle School at Durlach. The castle was extensively renovated from 1973 to 1988. Today, it houses the registry office, the Pfinzgau Museum, the Carpathian-German Museum, classrooms for adult education and the Margrave School, the historic banquet hall and the Durlach city library.