The Colts said they will hold a news conference at 3 p.m. ET on Thursday to introduce Pagano. That announcement came shortly after a league source told NFL.com's Steve Wyche about the move.
Debate: Pagano's in; what's next?
It's the third time Irsay has turned to a defensive-minded coach since replacing his father as owner in 1997, first hiring Jim Mora in 1998 and then Tony Dungy in 2002. Jim Caldwell, who succeeded Dungy in 2009 but was fired earlier this month, was a quarterbacks coach.
"Chuck is unorthodox," Suggs told The Associated Press. "He's like The Joker. You never really expect what he's going to do, and everything has a motive."
It will be the first head-coaching job for Pagano, 51, who just ended his first season as the Ravens' defensive coordinator after three years coaching their secondary. Baltimore finished the 2011 season as the NFL's third-ranked defense in both points allowed (16.6 per game) and total yards allowed (288.9 per game).
Pagano began his coaching career in 1984 as a graduate assistant at USC and spent time at Boise State, UNLV, East Carolina and Miami before joining the Cleveland Browns in 2001 to coach their secondary. He was the Oakland Raiders' defensive backs coach in 2005 and 2006, then served as defensive coordinator at North Carolina before joining the Ravens.
Pagano's Colts staff won't include his brother, John, whom the San Diego Chargers promoted to defensive coordinator earlier this month. John Pagano, an assistant in San Diego for 16 seasons, told the team's official website that while he's happy for his brother, "I have the job I've always dreamed of."
The Associated Press contributed to this report.