Edelman has 3 seasons with more than 150 targets. Those seasons are:
2013 - massive injury issues, particularly at the start of the season, double rookie lineup
2019 - AB gone/Sanu injured/Gordon gone/no Gronk, double rookie lineup in 2nd half
2016 - Floyd experiment, Gronk out for half the season, Amendola injured, Bennet & Hogan dinged up, rookie in lineup
So, no, it's not because Brady was doing some "circle of trust" thing. It's because those were years where the alternate targets sucked, got injured, or had to improve/adjust to the team and/or scheme.