Williams was born in Pibwr-lwyd, Llangynnwr, Carmarthenshire, Wales to a Methodist minister and his wife. He was educated at Carmarthen grammar school and Jesus College, Oxford (matriculated 1775, BA 1778, MA 1781). He was ordained deacon in 1777 and priest in 1778. He was a curate of Trelech, Carmarthenshire before becoming curate of Tetsworth, Oxfordshire. Williams became second master at the grammar school in Wallingford, Berkshire) and curate of the nearby village of Acton. Then, in 1780, he was made chaplain of HMS Cambridge under the command of Keith Stewart (son of the Earl of Galloway). He also tutored Lord Garlies, Stewart's nephew, who was a midshipman.
After a few years, the Earl of Galloway asked Williams to become the family tutor. He also assisted with research into the Galloway ancestry, helping to establish the Scottish Earl's claim to the English peerage. A Genealogical Account of Lord Galloway's Family was published in 1794.
In 1784, he was appointed vicar of Cynwyl Gaeo with Llansawel, Carmarthenshire. In 1799, he became curate of Chadwell St Mary, Essex and was also chaplain at Tilbury Fort. In 1809, while serving in Chadwell, he gave evidence in the case of White v Driver in which Elizabeth Manning's will was disputed on the grounds of insanity.
Whilst still at school, Williams helped with the 1773 publication of his father's Welsh bible and Mynegeir Ysgrythurol (Welsh concordance). As well as his genealogical research for the Galloway family, he published other genealogical works. He wrote Nautical Odes on the achievements of the Navy. After his death, his son from his second marriage published English works, which included essays on various Welsh and Celtic topics.
- Cooper, Thompson; Foster Evans, Dylan (2004). "Williams, Eliezer (1754–1820)". Oxford Dictionary of National Biography (online edition, subscription access). Oxford University Press. Retrieved 2008-02-26.
- Cases argued in English Ecclesiastical courts; White vs Driver