Take your guess when things started to go south between Brady's relationship with the Pats, but I'm going to guess it was after he was thrown under the bus for "Deflate Gate". In fact, I think it would've started even earlier had the defense failed on the final drive against Seattle in 2014.
After the 2017 Super Bowl is when things really started to go south as the offense put up 613 total yards and no puts yet lost. "ESPN's Ian O'Connor, in his book "Belichick," reported that Brady wanted a "divorce" from his coach. And Brady made it clear to author Mark Leibovich in the book "Big Game" that he was fed up with Belichick's culture, which is to say that he was fed up with Belichick. When asked in Leibovich's book how he would feel if the Patriots released him, Brady was blunt: "They can do whatever they want." They don't talk about it in the article, but remember when Gronk played games and considered retirement? That's why Bill traded him for a 1st round pick to Detroit. Gronk was pretty fed up with Bill as well.
In 2018, they gave Tom an incentive laden contract that put him in line with some pretty pedestrian QB's.
It's finally official: Tom Brady will not be forever a Patriot. But in reality, the quarterback's split with New England can be traced back much further than this offseason.
www.espn.com
By 2019, Tom had enough and was prepared to sit out the year if the Pats didn't remove the franchise tag. The most surprising thing was that Tom Brady at his age and accomplishments had negotiated to allow the team to tag him. Here's Hoyer on the situation:
During the Brown fiasco, it was Kraft that released him and not Bill. Brown asked Bill for another chance, but he told him he'd have to talk to Bob. It was clear Bob did this because he was already dealing with his own problems with the massage parlor incident.
Bob blames Bill for letting Tom go, but Bob is the one that could've easily stepped in and told Bill we are signing Tom to the two year $50M contract. The franchise was really starting to believe that Tom was just a product of it - their biggest mistake.