Pollitzer was born in Pest, Hungary. In 1842, he left Pest for Vienna, where he studied the violin under Joseph Böhm at the Vienna Conservatory; and in his 14th year he took the first prize at the Conservatory. After a concert tour in Germany, he went to Paris and studied under Jean-Delphin Alard. In 1850 he crossed the Channel, and in London his remarkable talents as a violinist were recognized after a short time. He became leader at Her Majesty's Theatre under Sir Michael Costa and also led the new Philharmonic Orchestra and the Royal Choral Society.
Pollitzer was a preeminent in his day as an interpreter of classic chamber-music, his playing attaining to what may be called "the great style". As a teacher of his instrument, he was regarded as the most eminent of his time in England and many pupils who attained distinction had studied under him. In 1861, on the establishment of the London Academy of Music, he was appointed professor of violin. He held this position until 1870, when he succeeded Dr. Henry Wylde as principal of the Academy and retained this position until his death, which occurred in London.
Pollitzer's pupils include Harold Bauer and Edward Elgar.