At a palace at the site, Lorenzo di Medici had used structures as a school and academy of arts (the Accademia degli Orti Medicei), where the likes of Pico della Mirandola, Lorenzo de Credi, Francesco Granacci, and Michelangelo frequented. When Piero de Medici was exiled in 1494, the villa was sacked.
In 1568-1574, the Francesco I de' Medici, Grand Duke of Tuscany and his potential half-brother Don Antonio de Medici commissioned Bernardo Buontalenti, to reconstruct the palace as a casino, a city villa, as it was sited at the Gardens of San Marco. Gherardo Silvani may also have played a role in the design. The palace was planned as a laboratory for scientific experimentation, in 1588 it became the seat of the Opificio delle Pietre Dure. On the death of Francesco I, in 1587 the palazzo was inherited by Antonio who took up residence here in 1597, commissioning numerous works for the rooms and garden which hosted sculptural groups by Giambologna.
The palace has been frescoed (1621–1623) by teams of artists including Anastasio Fontebuoni, Michelangelo Cinganelli, Fabrizio Boschi, Matteo Rosselli, Ottavio Vannini, Bartolomeo Salvestrini, Giovanni Battista Vanni, Jacopo Confortini, Domenico Pugliani and Jacopo Vignali. Filippo Tarchiani, in 1623, decorated the chapel with paintings depicting the Life of San Giuseppe (restored in 1967).
Inside the palazzo, don Antonio created a research laboratory, known as the Fonderia, and assembled various scholars interested in chemical and alchemy. The library is now hosted by the Biblioteca Nazionale Centrale di Firenze (BNCF).
After the Medici, the building played various roles, barracks, customs house, ministry of finances, finally court of appeals until 2012.
The exterior walls, windows, and portals are decorated with a whimsical array of eccentricities, a defining feature of Mannerist architecture.