Boylan began his coaching career as a player-coach in Switzerland from 1982–86 and led the Vevey Basketball club to its first championship in its 30-year history. At age 31, he moved back to the States becoming an assistant under Jud Heathcote at Michigan State University from 1986–89. Boylan then took over head coaching duties at the University of New Hampshire, succeeding Gerry Friel.
In 1992 Boylan entered the NBA as a video coordinator and advance scout for the Cleveland Cavaliers. In 1997 he moved over to the Vancouver Grizzlies, serving as an assistant under Brian Hill and later Lionel Hollins. In 2001, Boylan became a member of Frank Johnson's coaching staff in Phoenix, and remained working for the Phoenix Suns under Johnson's successor Scott Skiles. After Skiles was fired in 2002, Boylan worked with Terry Stotts for the Atlanta Hawks during the 2003-04 NBA season.
In 2004, Boylan became lead assistant to Skiles, who had taken over as head coach of the Chicago Bulls. On December 27, 2007, after the firing of Scott Skiles, Jim was named the interim coach for the Bulls for the remaining season. Boylan was not retained at the conclusion of the season after compiling a 24–32 record with the Bulls. On May 14, 2008, he was hired as an assistant to Scott Skiles by the Milwaukee Bucks. When Skiles resigned in January 2013, Boylan became head coach of the Bucks. The team went 22–28 under his guidance and made the playoffs, but were swept in the first round by the Miami Heat. At the end of the season, the Bucks decided not to give Boylan a new contract.