Well in CLE he took over a bad team that he gradually improved to the point of a playoff team with year-over-year progression. Eventually he got them to the playoffs. Then the bottom fell out the last year with the team moving and all. Over his first 3 years in CLE he had a losing record each time, but the overall direction was positive culminating in that great year 4 with a playoff win. He took over a 3-13 team and not everything is a quick turnaround. It was a slow build for him there. I don't hold that losing record against him for the first 3 years. The last year I chalk up to being a pretty tough situation to coach through with the team moving. They finished 5-11 but they actually started 3-1 and were still .500 when the owner announced the sale. That goes on his record as a losing season just the same though.
In NE, his first year here, he gutted the roster. They played parts of the year with less than 53 guys to help get the cap in order. Similar to CLE, he had a multi year plan in place and year 1 involved taking a step back to go forward. I don't fault him too much for that. The next year he started 0-2 with Bledsoe which is a blemish and definitely makes you ask the question of how much Brady was responsible, even at that stage of his career where he probably wasn't as good an overall player. It was 0-2 with 2 one-score losses, so not exactly like they were incompetent until Brady solved everything, but yeah it's question.
In his later NE years, he did an absolutely terrible job. That's why his record sucked. He was an awful GM over those last few years and the years preceding that. He made it worse by refusing to acknowledge it and properly rebuild like they needed. He chased stop gap QBs in the older, washed up Newton and low ceiling but NFL ready Mac Jones. He refused to go for the multi year rebuilds he did in CLE and NE and it left the roster mired in mediocrity until the bottom fell out his last year. His coaching circle dried up because he's 70+ years old and he didn't have a circle of trust to pull from because at a certain point they stopped developing coaches.