Brown, a guest speaker at Thursday's Santa Clara (Calif.) Sports Law Symposium, said he will join NFL Commissioner Roger Goodell this weekend at a meeting with the Congressional Black Caucus in Washington rather that fly to Ohio to participate in the ceremony honoring Cleveland's 16 Hall of Fame members.
"No, I won't be going," Brown said. "My legacy won't be defined by Holmgren."
Brown is upset with Holmgren for reducing his responsibilities as an adviser to team owner Randy Lerner. Brown sent a letter telling Holmgren he wouldn't stay with the team to be a "greeter, that of a mascot."
Holmgren attempted to contact Brown and wrote a letter trying to convince the 74-year-old NFL legend to change his mind, but was unsuccessful. Holmgren said Lerner also sent a letter to Brown trying to persuade him to attend the ceremony Sunday when the Browns will honor their initial class into the ring of honor.
"I haven't heard from Jim, but the door is open," Holmgren said earlier this week when the Browns kicked off events connected with their ring of honor. "After he declined, I wrote another letter to him and again expressed my feelings on how much we would like him there, but understood that he might not be able to come.
"I left the door open. I'm holding out hope that we get that call in the next couple days."
At the symposium, Brown also said he isn't surprised by the rise in diagnosed concussions among NFL players and believes the league and the union need to do more to protect those players. He added that players must be better educated about their own health so they don't attempt to hide the injuries.
Brown wasn't the keynote speaker at the event, but he was easily the most popular and recognizable person at the front table. His message was clear: Professional sports leagues must improve their care and education of the athletes.
Dr. Hunt Batjer, co-chairman for the NFL's Brain, Head and Neck Medical Committee, said earlier this week he didn't see the concussions suffered by the four players as part of a trend, adding that the league and its medical staff will closely monitor the situation through the season.
But Brown, a longtime activist and proponent for change in the health care of current and former NFL players, believes the concussions are symptoms of a larger issue.
"Concussion have brought the consciousness to the problem, but I think the problem is football-related injuries period and the lack of support from the league of those players who have suffered those injuries," Brown said. "The denial factor has been unbelievable. I'm here because I'm a fighter to try to bring attention to this fact."
Specifically, Brown blasted the NFL for often turning a blind eye to head injuries suffered by players. The league, he said, promotes hard hits but doesn't do enough to deal with the ramifications.
"It doesn't take science to know that when you have head-to-head collisions, there's going to be some effect," Brown said. "Boxing is a great example of it, but in football, sometimes you're taking greater hits than boxers. When you have one man going full speed against another man and those heads are colliding, it's just the fact of science you're going to have results.
"All the denial that's taken place over the years to keep the league from having to pay money or the players association taking advantage of their players and not representing them properly, all those things have gone on. Only now years later, here we are saying concussions. People have been getting knocked out for years and going back in the game unsupervised."
Brown lauded the NHL for its efforts in diagnosing and treating head injuries and said the NFL needs to follow suit.
"People want football and they want hard-hitting football, so to me, it's not the thing of hard-hitting football," Brown said. "It's at least taking care of your wounded. I don't want football to not be played, but I would like the sophistication brought forth to take care of those who need to be taken care of and to take the precaution, at the sacrifice of winning, to take care of people."
This past June, Brown was presented with the Blanton Collier Award by the Kentucky chapter of the NFLPLA to acknowledge his humanitarian work. Brown's Amer-I-Can organization helps gang members from inner cities move toward a more productive life, and he also is involved with numerous other charities.
Brown's current passion, however, is forcing change in the NFL on numerous levels. He talked about the need to revamp the league's pension plan and health-care system, as well as a rookie salary cap.
Historically, Brown said, the NFL has looked the other way when players suffer concussions in order to keep the player on the field despite the increased risk of injury.
Kolb was hurt shortly after Bradley was, though the Eagles originally reported it as a jaw injury. Like Bradley, Kolb briefly came back in the game before heading to the sideline.
Part of the problem, Brown said, is the lack of education among athletes.
"Players have to recognize when something is wrong and stand up," Brown said. "That's something that's going to be difficult because players are ostracized when they do that ... so there's a tremendous sacrifice that goes inherently with the game. That's not anybody's fault but the players themselves because we should be able to say, 'Hey, I don't feel good. I can't play."'
The Associated Press contributed to this report.