Many people were surprised when Oliver became one of the first five-star recruits to sign with a non-Power Five conference program. The Cougars had signed his brother (Marcus, a two-year starter at right guard) and former high school coach, however, so E.J. (short for Ed Jr.) signed to play near his hometown of Westfield. He was a second-team All-American and top-5 overall recruit who eschewed Alabama to play for UH. He fulfilled his promise quickly, earning first-team All-American and all-conference honors as a true freshman, starting all 13 games and ranking second in the country with 23 tackles for loss, including 3.5 against San Diego State in the Las Vegas Bowl. He also was credited with five sacks among his 66 total tackles, as well as three forced fumbles and nine pass breakups. Oliver won the Outland Trophy and was a finalist for the Nagurski Award as a sophomore, also garnering consensus All-American honors and winning the American Athletic Conference's Defensive Player of the Year Award with 73 tackles, 16.5 for loss, 5.5 sacks, and three pass breakups despite facing constant double-teams. He struggled with a knee injury throughout his junior season and had a televised blow-out with third-year head coach Major Applewhite (who was fired after the season) over whether he should be wearing a heavy jacket on the sideline meant for suited players while nursing that injury. In the end, he started eight games (54 tackles, team-high 14.5 for loss, three sacks, two pass breakups) and earned second-team All-AAC honors and third-team All-American notice from the Associated Press. Oliver's father was a running back at Northwestern State in Louisiana with LSU head coach Ed Orgeron and NFL players Gary Reasons and Mark Duper.