The team announced Tuesday that it had fired Caldwell, the second major decision in what is expected to be a long offseason shake-up. Colts owner Jim Irsay fired vice chairman Bill Polian and his son, general manager Chris, one day after the team finished a 2-14 season in which Manning didn't play because of neck surgery.
Caldwell led the Colts to a 14-2 regular-season record and a Super Bowl appearance in the 2009 season, his first after being hand-picked by Tony Dungy to take over the head-coaching duties. Caldwell was the Colts' quarterback coach/assistant head coach from 2002 to 2008.
By winning his first 14 games, Caldwell set an NFL record for a rookie coach, and he became just the fifth first-year coach to take his team to the Super Bowl, although the Colts lost to the New Orleans Saints in that game.
"This was a difficult decision," Colts owner Jim Irsay said during a Tuesday news conference. "It was based on the feeling that this was a direction the franchise needed to go. I wanted to make sure we took all the time we needed to make sure it was the right decision."
With the Colts' head-coaching position now open, there could be speculation that the team might approach Dungy about a possible return. But NFL Network insider Jason La Canfora said Tuesday on "Around The League" that there's no reason to believe Dungy could return to the coaching ranks.
"Everything I've heard is he's perfectly happy doing what he's doing," La Canfora said of Dungy, who has been working for NBC's "Football Night in America" since leaving the coaching ranks. "There's really been no smoke there for a couple of years now. He's been pretty equivocal with his stance on this."
Newly hired general manager Ryan Grigson said the Colts will be patient during their search for a coach.
"We don't want to do anything rash in any regard with any of this," Grigson said. "These are big decisions, and we don't want to be hurried."
Grigson said Caldwell was informed Tuesday of the team's decision to move in another direction.
The Associated Press contributed to this report.