The contract is for seven-years and worth $35 million, including $10 million guaranteed, reports NFL Network's Adam Schefter. The contract numbers are not final, but the two sides were discussing a deal that would average $5 million per year and pay $10 million in guaranteed money.
Cleveland does not have any draft picks in the first three rounds of April's NFL Draft, so it is stockpiling players through free agency.
The Browns have been one of the most active teams in the league over the first 48 hours of the free agent signing period. In addition to Stallworth, the Browns have also traded for defensive tackles Shaun Rogers and Corey Williams and re-signed Anderson and running back Jamal Lewis.
The 6-foot, 200-pound Stallworth has played six years and 84 games in the NFL with New Orleans, Philadelphia and the Patriots. He has 279 receptions for 4,213 yards and 31 touchdowns.
Last season with New England, Stallworth had 46 receptions for 697 yards and three touchdowns. He had a career-high 70 receptions and 945 yards with New Orleans in 2005.
"Donte' will be the major addition to our offense this year," Browns GM and senior vice president Phil Savage said. "He will bring legitimate speed, quickness and playmaking ability to the receiving corps and add a different dimension to the skill players we already have in place."
Stallworth was taken in the first round, 13th overall, by New Orleans in the 2002 draft. In his rookie season with the Saints, he had a career-best eight touchdown receptions.
The Associated Press contributed to this report