The team later confirmed the report.
"This is the worst part of this business," said John Elway, Denver's executive vice-president of football operations.
Xanders, 41, spent four years in the organization, three as the team's general manager. He had one year left on his contract.
"John and I had a lengthy conversation today and we mutually decided to part ways for the best interest of my career aspirations," Xanders said.
The move comes following a Monday morning meeting with Elway.
"It was tough because I have a great deal of respect for Brian as a football man but also as a person," Elway said. "Even more so as a person. He was very helpful to me in this first year getting acclimated back to the NFL. I owe him a lot. Just from the conversation we had, we just thought for his career and for the Broncos, we felt that now was the best time to go a different way."
"Brian deserves a tremendous amount of credit and recognition for the contributions he made to the Broncos during his four years with the organization. His hard work and dedication played a major role in the recent success of our team, most notably last season's division title and playoff win," Elway said.
Xanders spent 14 years with the Atlanta Falcons in a variety of capacities before being hired by the Broncos in 2008 as assistant general manager by Mike Shanahan, head of football operations at the time. The next year, Xanders was promoted to general manager by then head coach and head of ootball operations, Josh McDaniels.