You can't "take away" any one game that "skews" the stats, or you'd have to do that for everyone. So what you do is calculate how the Pats' opponents have fared against their non-Patriots' opponents (on average), and then compare. So, through four games....
Houston
vs. NE: 20 points, 325 yards, 21 first downs
vs. other: 25.3 points, 443.3 yards, 24.0 first downs
DIFF: -5.3 points, -118.3 yards, -3.0 first downs
Jacksonville
vs. NE: 31 points, 480 yards, 27 first downs
vs. other: 19.0 points, 346.7 yards, 17.3 first downs
DIFF: +12.0 points, +133.3 yards, +9.7 first downs
Detroit
vs. NE: 26 points, 414 yards, 25 first downs
vs. other: 22.3 points, 382.7 yards, 21.0 first downs
DIFF: +3.7 points, +31.3 yards, +4.0 first downs
Miami
vs. NE: 7 points, 172 yards, 11 first downs
vs. other: 25.0 points, 324.0 yards, 16.3 first downs
DIFF: -18.0 points, -152.0 yards, -5.3 first downs
NET DIFF: -7.6 points, -105.7 yards, +5.4 first downs
So they're better than average in points and yards, and worse than average in first downs. That sounds pretty much like a team whose defense should be ranked somewhere between 12th and 14th.
Houston
vs. NE: 20 points, 325 yards, 21 first downs
vs. other: 25.3 points, 443.3 yards, 24.0 first downs
DIFF: -5.3 points, -118.3 yards, -3.0 first downs
Jacksonville
vs. NE: 31 points, 480 yards, 27 first downs
vs. other: 19.0 points, 346.7 yards, 17.3 first downs
DIFF: +12.0 points, +133.3 yards, +9.7 first downs
Detroit
vs. NE: 26 points, 414 yards, 25 first downs
vs. other: 22.3 points, 382.7 yards, 21.0 first downs
DIFF: +3.7 points, +31.3 yards, +4.0 first downs
Miami
vs. NE: 7 points, 172 yards, 11 first downs
vs. other: 25.0 points, 324.0 yards, 16.3 first downs
DIFF: -18.0 points, -152.0 yards, -5.3 first downs
NET DIFF: -7.6 points, -105.7 yards, +5.4 first downs
So they're better than average in points and yards, and worse than average in first downs. That sounds pretty much like a team whose defense should be ranked somewhere between 12th and 14th.