He was born at York in 1726 and entered Jesus College, Cambridge, in 1745. He succeeded his father in 1742, inheriting Birdsall House. In 1757, he served as High Sheriff of Yorkshire. He succeeded his cousin Thomas Willoughby to the Middleton barony and estates (Wollaton Park and Middleton Hall, Warwickshire) in 1781.
- Henry Willoughby, 6th Baron Middleton (1761–1835)
- Hon. Dorothy Willoughby (d. 13 April 1824, married on 24 November 1784 Richard Langley of Wykeham Abbey
- Hon. Henrietta Willoughby (d. March 1846), married Richard Lumley-Saunderson, 6th Earl of Scarbrough
His memorial is in St. Leonard's Church, Wollaton.