Tengbom was born in Vireda in Jönköping County, studied at the Chalmers School of Technology in Gothenburg 1894-1898, at the architecture school of the Royal Swedish Academy of Arts in Stockholm 1898-1901 (being awarded the so-called Royal Medal) and abroad 1905-1906. He worked 1906-1912 with Ernst Torulf in Stockholm and Gothenburg 1906-1912, and on his own from 1912 in Stockholm. He was appointed architect in the Office of the Chief Intendant in 1906 and professor of architecture in the Royal Swedish College of Art in 1916. He became a member of the Royal Swedish Academy of Arts in 1917. In 1921 he was appointed Director General of the State Office of Construction (Byggnadsstyrelsen).
The architect firm Tengbom & Torulf won second prize in the 1905 competition for the Stockholm City Hall building (after Ragnar Östberg), and in 1906 again second prize for the Engelbrektskyrkan (Engelbrekt Church) in Stockholm (built according to the design of Lars Israel Wahlman). They were more successful in the competition for the City Court building (rådhus) in Borås in 1909, where they won first prize and were allowed to execute their design. Another public building designed by Tengbom in collaboration with Torulf was the new church in Arvika, completed in 1911. The two also received the commission to build a hunting lodge for Eric von Rosen in what is today Jaktstuguskogen Nature Reserve. The Trelleborg Water Tower was built after drawings by Tengbom and completed in 1912.
After Tengbom left the collaboration with Torulf, he made the design for the main office of the Stockholms Enskilda Bank at the Kungsträdgården Park in Stockholm (1912–1915). Another Stockholm office for the bank, at Götgatan on Södermalm, was built according to Tengbom's design in 1916. Another bank office was the one designed for the Borås Enskilda Bank (1916). Other Tengbom buildings from the time period were that of the building for the daily newspaper Svenska Dagbladet at the street Karduansmakargatan in Stockholm, and the Högalidskyrkan (Högalid Church) in Stockholm (after winning first prize in a competition).
In the 1920s he made the design for the building of the Stockholm School of Economics (1925) and the Stockholm Concert Hall (1923–1926) at Hötorget Square. The home of the Royal Stockholm Philharmonic Orchestra and known as the place where the Nobel Prize ceremony takes place, the Hall, a light-blue building with a portico with tall, slender polygonal columns with corinthian capitals. The concert hall is probably Tengbom's best-known building and, together with Gunnar Asplund's Stockholm Public Library, the most widely recognized example of the neo-classical architecture of the Swedish 1920s, in English referred to as Swedish Grace.
In the last years of the 1920s, he designed the main office of Ivar Kreuger's corporation Svenska Tändsticksbolaget at Trädgårdsgatan in Stockholm. His later production includes the building for the Swedish Institute at Rome 1938-1940. He was awarded one of the inaugural Prince Eugen Medals in 1945 for architecture.
His son Anders Tengbom (1911–2009) was also, in his own right, a very famous architect. One of his greatest creations was Bonnierhuset, one of the tallest buildings in Stockholm. His daughter, Ann-Mari Tengbom, married Otto Christian Archibald von Bismarck.