He was born in Philadelphia, the son of Russian immigrants Lillian and Louis Kosloff. He had two brothers, Carl and Charles Kosloff. He graduated from South Philadelphia High School in 1930, where he played football and basketball as an offensive center and middle linebacker. Initially, Irv set out to be a dentist. However, after a knee injury, he lost his athletic scholarship to Temple University, which he had attended for approximately two years but had to drop out and search for a job for lack of funds. He found a job in the traffic department of the Container Corporation of America in Manayunk, Philadelphia. In 1932, he founded the Roosevelt Paper Company on State Road near Cottman Avenue in Philadelphia. The company grew to become one of the largest paper merchants in the country.
In May 1963, Kosloff, with the inspiration and aid of high-school classmate attorney Ike Richman, purchased the Syracuse Nationals of the National Basketball Association from Danny Biasone, brought them to Philadelphia, and changed the team's name to the Philadelphia 76ers. Richman ran the day-to-day operations of the team, with Kosloff, busy with his growing paper business, remaining the silent partner.
During Kosloff's tenure as owner, the Sixers won an NBA Championship in 1966-67.