Goulburn was the third son of Munbee Goulburn of Portland Place, London, and his wife Susannah. Frederick was the younger brother of Henry Goulburn, under-secretary for the colonies 1812–1821 and later Chancellor of the Exchequer. Frederick Goulburn joined the army as a cornet in the 23rd Light Dragoons on 25 May 1805. Goulburn transferred to the 13th Light Dragoons in June 1813 and fought in the Peninsular War and later at Waterloo.
Goulburn was appointed secretary and registrar of the records of New South Wales on 30 June 1820, very likely due to his brother's patronage. He was sworn-in as Colonial Secretary on 1 January 1821.
Also in 1821 Thomas Brisbane was appointed Governor of New South Wales. He was doing useful work, but he could not escape the effects of the constant factional fights which also plagued previous governors. Brisbane discovered that Goulburn had been withholding documents from him and answering some without reference to him. In 1824 Brisbane reported his conduct to Lord Bathurst who was Secretary of State for War and the Colonies. In reply, Bathurst recalled both Brisbane and Goulburn in dispatches dated 29 December 1824.
- Parsons, Vivienne. "Goulburn, Frederick (1788–1837)". Australian Dictionary of Biography. Canberra: Australian National University. Retrieved 16 September 2014.
- "The Colonial Secretaries, 1821-1959". New South Wales Government State Records. Archived from the original on 21 May 2013. Retrieved 12 August 2013.
- "Government and general orders". The Sydney Gazette and New South Wales Advertiser. National Library of Australia. 6 January 1821. p. 1. Retrieved 16 September 2014.
- "Mr Frederick Goulburn (1788-1837)". Former Members of the Parliament of New South Wales. Retrieved 18 April 2019.
|New South Wales Legislative Council|
|New creation|| Nominated member
December 1823 – December 1825
|New creation|| Colonial Secretary
of New South Wales
1 Jan 1821 – 7 Jan 1826