Historically Chivery was part of the manor of Aston Clinton. It was created at the end of the 12th century when William de Clinton created the manor of Aston Chiverey provided a dowry for his daughter Alice who married Reginald de Mohun. On his death in 1213 Alice married Robert de Beauchamp. From 1250 until around 1380 the manor was held by the Audley family until through marriage it passed to John Rose, an esquire of Richard II of England. By 1410 in Henry IV's reign it had passed to the Crown before being recovered by John Rose's descendant Thomas St Clair. Thomas' lands were divided on his death in 1483 and his daughter Eleanor inherited the manor of Aston Chiverey and through marriage the land became part of the estates of John Gage. His grandson Sir John Gage sold the manor to Margaret Pole, 8th Countess of Salisbury in 1534, whereon it became recombined with the manor of Aston Clinton once more. The Countess was executed on Henry VIII's orders in 1451 and all her lands were forfeit and split up. Ownership of the manor resided with William Batchelor up until 1555 when the properties passed in his will to his brother-in-law Thomas Gewat. However by 1582 the fields were being farmed by Silvester and John Baldwin and the woodlands by another John Gage.
By this time also the manor had become part of the St Leonards parish, which had split from Aston Clinton. During the 19th century Chivery became part of a new parish of Aston Clinton St Leonards.In 1934 Chivery was hived off from St Leonards which was combined with other parishes to become Cholesbury-cum-St Leonards parish within Amersham Rural District while Chivery became part of Aston Clinton parish part of Aylesbury Rural District.
From 1776 Chivery had a pub, The Plough which survived into the 20th century becoming a restaurant before finially closing in the 1930s.