Williams lettered in basketball and baseball at T. C. Roberson High School outside of Asheville, North Carolina and went on to play basketball at UNC under legendary coach Dean Smith. After graduating with a bachelor's degree in education in 1972 and an M.A.T in 1973, Williams went on to coach high school basketball and golf at Charles D. Owen High School in Swannanoa, North Carolina. In 1978, Williams came back to UNC and served as an assistant to Coach Smith. During his tenure as assistant coach, UNC went 275-61 and Williams played a role in Michael Jordan's recruitment. In 1988, Williams left UNC and replaced former UNC assistant Larry Brown as the head coach of the University of Kansas Jayhawks.

After averaging about an 80% win percentage in 15 seasons at the University of Kansas, he became the eighteenth head coach at North Carolina when he replaced Matt Doherty in 2003. He received his 400th win in January 2003, when Kansas beat the University of Wyoming. On April 4, 2005, Williams shed his title as "the best coach to never have won a NCAA ring" as his Tar Heels defeated the University of Illinois Fighting Illini in the 2005 NCAA championship game.

Williams has won the AP Coach of the Year award in 1992 with the Kansas Jayhawks and again in 2006 with the North Carolina Tar Heels. He is only the seventh coach in history to win the award twice and the second to do it at two different schools.

In 2009, Williams again led the Tar Heels to the National Championship over Michigan State. In 2016 his UNC team made it to the NCAA Championship game only to lose at the buzzer to Villanova.