Born Charles FitzRoy, he was the illegitimate son of Charles FitzRoy, 2nd Duke of Grafton and educated at Westminster School from 1721 to 1730. He married Frances Scudamore in 1744 after her divorce from Henry Somerset, 3rd Duke of Beaufort, in 1743. She was the only child and heir of James Scudamore, 3rd Viscount Scudamore, bringing him the Viscounts Scudamore seat of Holme Lacy. Fitzroy added the Scudamore name to his own on 22 March 1749.
He was Member of Parliament for Thetford (1733 to 1754), Hereford (1754 to 1768), Heytesbury (1768 to 1774) and Thetford again from 1774 to March 1782. Due to his continued forty-eight-year service in the British House of Commons, FitzRoy-Scudamore succeeded William Aislabie as Father of the House in 1781 but died a year later.
His only child, Frances (1750–1820), became the second wife of Charles Howard, 11th Duke of Norfolk, but became insane and was locked away for many years. After her death without children, the estate of Holme Lacy fell into extensive litigation, eventually settling on Sir Edwin Stanhope, 3rd Baronet, who adopted the additional surname Scudamore.
- Sedgwick, Romney R. (1970). "FITZROY (afterwards FITZROY SCUDAMORE), Charles (?1713-82), of Holme Lacy, Herefs.". In Sedgwick, Romney (ed.). The House of Commons 1715-1754. The History of Parliament Trust.
- Burke's Baronetage