In 1954, Levy joined the coaching staff at the University of New Mexico and was named head coach in 1958. In two seasons as head coach, he guided the Lobos to a 14-6 record and earned Skyline Conference Coach of the Year honors both years. He finished his college-coaching career with a five-year stint as head coach at the College of William and Mary where he twice earned Southern Conference Coach of the Year honors.

Levy began his professional football coaching career in 1969 as kicking teams coach for the Philadelphia Eagles before joining George Allen's staff as a special teams coach for the Los Angeles Rams in 1970.

In 1987, his first full season with the Bills, the team returned to respectability with a 7-8 record and were in the playoff hunt throughout most of the season. The following season the team posted a 12-4 record and won the first of six AFC Eastern Division titles. With his high-powered "no-huddle" offense, Levy went on to set a new standard for innovation among NFL coaches as he led his AFC championship team to four consecutive Super Bowl appearances.

From 1988 through 1997, the Bills were first in the AFC in winning percentage and second only to the San Francisco 49ers in the NFL. Levy, the winningest coach in Bills' history, recorded a 112-70 regular season record and was 11-8 in the playoffs during his eleven seasons with the Bills. He was named NFL Coach of the Year in 1988 and AFC Coach of the Year in 1988, 1993, and 1995.

Levy retired in 1997 and became an analyst for NFL.com.

In 2001, Levy was elected to the Pro Football Hall of Fame in Canton, Ohio along with former Bills' special teamer Steve Tasker.