"I have no interest in talking to Gregg," Payton said of his former defensive coordinator, and he might not be alone.
Williams, still suspended indefinitely by the NFL for his role in the bounty scandal, has been given permission by the league to seek work. Finding it might be another issue.
Jason Cole of Yahoo! Sports talked to a handful of coaches around the league who question whether Williams still has a place at the table.
"Fabulous coach, energetic, inventive, aggressive ... but after all the stuff that happened, are your players going to worry about what they do or say in front of him?" one head coach told Cole. "Heck, the way you hear it from the people (in New Orleans), he won't listen to superiors."
From a second coach: "I've been around Gregg a lot over the years and I like him, but he has always been a guy who pushed it to the limit as much as possible."
Williams finds himself in an awkward spot. Any job offer he receives must first be reviewed by NFL Commissioner Roger Goodell, who has yet to lift Williams' suspension. That's an extra hoop for suitors to jump through when the weighty baggage on Williams already makes this a tough hire.
Follow Marc Sessler on Twitter @MarcSesslerNFL.