My first three modifiers for points-allowed are:
1. Pace. How many drives per game did each side have? I'd conjecture that the Pats had relatively few possessions per game early in the BB/Brady era, but have been on the high side in recent years.
Note that drives/game is the important metric, not plays, although the latter has a lot to do with the former.
2. Offensive TOs. If an offense frequently turns over the ball deep in its own territory, the defense can be excused for giving up more points. Since it's hard to get stats that measure just TOs leading to bad field position, it's natural to look at all TOs as a proxy.
The Pats' defense has had an easy time by this metric during the Brady years.
3. Defensive TOs. Defenses actually have two jobs. The more important is preventing scoring. The other is getting the ball with good field position. Some defensive TOs have that effect.
If memory serves, the defense has been quite up-and-down by this metric.