James, a running back, attended Southern Methodist University where he and teammate Eric Dickerson formed what was known as the "Pony Express" backfield (SMU's mascot is the mustang). In the 1980 Holiday Bowl, James' offensive production of 225 rushing yards and 9.9 yards/carry were both Holiday Bowl records that stood for 13 and 15 years respectively. In 1982, he scored on a 96-yard touchdown reception to set a new record as the longest scoring play in Southwest Conference history.

He graduated in 1983 and went to play in the USFL for one year, after which he joined the New England Patriots. He was named Offensive Player of the Year by the Vince Lombardi Committee in 1985 and started in both the 1985 Pro Bowl and Super Bowl XX. In the 1985-86 season, James rushed for 1,227 yards.

After his retirement from playing in 1989, James worked much of the time in radio and television broadcasting calling football games and being a football television analyst for CBS and ABC.