"I don't even know if stunned would be a strong enough word," Payton said Tuesday, referring to what his reaction would be if Brees left. "I don't see that happening."
Brees has thrown for more yards than any NFL quarterback since joining the Saints in 2006 and passed for an NFL record 5,476 yards this season, breaking Dan Marino's 27-year-old mark of 5,084.
The star quarterback was among a number of players who had already left town and were not available for comment on Tuesday, the first day Saints headquarters was opened to reporters since last weekend's upset loss to San Francisco in the divisional round of the playoffs.
Receiver Lance Moore said he was already operating under the assumption Brees would be his quarterback in 2012.
"As important as he is not only to this team but this city, this whole area, you kind of have to do right by him and sign him to a long-term deal because he's earned that," Moore said. "Where would we be without Drew? You can't even fathom that."
The Saints gained an NFL record 7,474 offensive yards this season, easily eclipsing St. Louis' mark of 7,075, which had stood since 2000.
"This season, certainly, there are some real good players that potentially are free agents," Payton said. "Each year those are challenges and that's what makes it hard to win in our league. ... We'll look closely at it and try to be objective."
Other free agents include cornerback Tracy Porter, defensive tackles Aubrayo Franklin and Shaun Rogers, linebackers Jo-Lonn Dunbar and Ramon Humber, and reserve defensive ends Jeff Charleston and Turk McBride. Starting center Brian de la Puente, receiver Adrian Arrington and backup quarterback Chase Daniel are restricted free agents.
Payton said it was apparent shortly before the season ended that Williams, with his contract expiring, was likely going to join Fisher, an old friend under whom he coached for in Tennessee for a decade. Payton said the Saints and Williams never discussed an extension.
Payton said he'll keep an open mind about assistant coaching candidates and did not expect to rush into any hirings. In addition to defensive coordinator, there is also a wide receivers coach opening because of Curtis Johnson's earlier decision to become head coach at Tulane after the Saints' 2011 campaign ended.
"Real talented coaches come in all age groups and they come from all over. They come from college, they come from our league. Some have been head coaches, some haven't," Payton said. "I just look back on my own experiences. Someone gave me an opportunity for the first time. So you try to pay attention to all those things."
Throughout the locker room, players said they were still trying to get over the shock of their dramatic loss and stomach-turning thoughts of a missed opportunity - because of the Giants' upset of Green Bay - to host an NFC title game in the Superdome, where the Saints were unbeaten this season for the first time in franchise history.
"I can almost guarantee you that nobody around here thought it would be over this soon," Moore said. "Right now our main focus is just to kind of get away and not think about it too much because it'll drive you crazy."
At some point, several players said, they'll fondly remember a season that was remarkable in many ways, even if it ended before they thought it would.
"Down the line I'll look back on this and know that it was a special year," said de la Puente. "We did have a very, very good year - won the NFC South, broke all kinds of records, our offense was awesome. You can pick and choose how you want to look at the season ... but I think if you can keep the core guys together, this team has a very positive future."