Born in Kexborough near Barnsley, Driver studied at the University of Sheffield, where he edited the student newspaper, ran the Socialist Club, and joined the Communist Party of Great Britain (CPGB). After university, he returned to Kexborough, where he picked potatoes and ran the local Labour Party. In 1936, he married Thora Senior, the couple having two children. Driver became a French teacher in 1937, initially at Barnsley Central School, then at Keighley Junior Technical School. He also became active in the National Union of Teachers (NUT).
In 1947, Driver moved to work at the new Doncaster Technical College. He joined the Association of Teachers in Technical Institutes (ATTI), and was elected as its national president in 1961, also remaining a member of the NUT, and winning election to its national executive. At the time, no teachers' unions were affiliated to the Trades Union Congress (TUC), but Driver consistently spoke in favour of their affiliation. He persuaded the NUT to join the TUC in 1969, and it was gradually followed by the other main unions of teachers.
In 1969, Driver was elected as general secretary of the ATTI. He believed that there should be one union representing college lecturers, and in 1976 he completed a merger with the Association of Teachers in Colleges and Departments of Education, forming the National Association of Teachers in Further and Higher Education. Teacher training colleges at the time were suffering from job losses, and Driver therefore spent much of the next couple of years supporting redundant lecturers in teacher training.