A few stats:
1. Since 2001 (I think) we have only swept the division twice. We went 4-2 seven times.
2. We have almost the same winning percentage inside the division (.777) as we do outside (.766) -- this link compiles that 2002-2016 but my guess is it's similar if you include the most recent two years:
3. Since 2013 we are 12-0 against the AFC North including 2 playoff games.
4. In the Brady era, the AFC East has been the best division in football by winning percentage. That of course is because the Patriots are in this division. But if you remove the Pats, and then (to make it fair) you remove the top team from every other division, and calculate the winning percentage of the bottom three teams, the AFC East is fifth out of eight. Not great, but not horrible.
5. The Dolphins and Jets have been right around .500 outside of their division (2002-2016), which makes them perfectly average. The Bills of course have been much worse, but every other division also has had a horrible team in that span, and if you rank the worst team in each division against each other, the Bills are actually the best!
Conclusions:
The Patriots would still have the same amount of success regardless of what division they are in. The theory of them having a huge advantage by playing in the AFC East is a myth.
The AFC East looks much worse by virtue of having to play the Patriots all the time. They also probably tailor their roster composition around trying to beat the Patriots, which is probably a suboptimal strategy.
The only good stat as a counterargument is the number of wildcard appearances per division, where I think the AFC East is last. This suggests that although the rest of our division isn't horrible (given the .500 record outside of the division), there also haven't been too many great teams to challenge us from time to time. The other divisions seem to cycle through each team going from great to horrible, while ours just stay consistently mediocre.