Baillie was the third son of Evan Baillie of Dochfour and his wife Mary Gurley, daughter of Peter Gurley of St. Vincent. His father was a landowner in Scotland with commercial interests in Bristol including the firm of Evan Baillie, Sons & Co. Baillie, together with his older brother Hugh Duncan Baillie, became a partner in the Bristol Old Bank in 1812 after death of his brother Peter Baillie. In 1813 Baillie became Member of Parliament for Tralee and held the seat until 1818. Baillie was a partner in J E Baillie, Fraser & Co of Bristol, Chairman of British Guiana Association, president of Whig Anchor club of Bristol and a Member of Brook's club. He was put up for parliament at Bristol without his consent in 1820 but in fact his brother Hugh stood unsuccessfully. He became Member of Parliament for Bristol in 1830 and held the seat until 1835. In parliament he was in favour of parliamentary reform and catholic relief.
Baillie continued to buy land in Scotland after his father's death in 1835 and bought Glentrome in Badenoch in 1835. He had been a prominent opponent of abolition but after abolition received a slave compensation of £53,964 in 1835–1836. When the British government emancipated the slaves in the 1830s, James and his brother Hugh received compensation for more than 450 slaves across 21 estates in British Guiana, Grenada, St Kitts, St Vincent, and Trinidad. This money was invested in further Scottish estates at Glenelg in 1837, Glenshiel in 1838 and Letterfinlay in 1851.
Baillie never married and died at the age of 82
- Joseph Gaston Baillie Bulloch. A history and genealogy of the family of Baillie of Dunain, Dochfour and Lamington : with a short sketch of the family of McIntosh, Bulloch, and other families.
- R. G. Thorne The House of Commons, 1790-1820, Volume 1
- "Leigh Rayment Commons constituencies T Part 2". Archived from the original on 15 July 2018. Retrieved 5 August 2010.CS1 maint: unfit URL (link)
- "Leigh Rayment Commons constituencies B Part 6". Archived from the original on 10 August 2009. Retrieved 5 August 2010.CS1 maint: unfit URL (link)
- Douglas Hamilton Scotland, the Caribbean and the Atlantic world, 1750-1820
- "Summary of Individual | Legacies of British Slavery".
- Hansard 1803–2005: contributions in Parliament by James Evan Baillie