To me drafting Harry instead of Brown was the the event that was ultimately BB's undoing with the Pats. Harry being so bad alienated Brady and led to the Sanu trade and other desperate decisions that helped create a cap space mess and push TB12 out the door. If AJ had been the choice, success on the field would have gone a long way towards Brady wanting to finish his career in NE. Brady may still have left for other reasons, the decision to leave would have been much harder if the Pats were still competitive.
Because BB went against the advice of the scouts he lost a lot of credibility and started the discussion that the game had passed him by. When he doubled down with the obviously bad plan to make MP the OC, his credibility was gone and he needed to go.