By DVOA, not even true. Unadjusted pass defense VOA was 4th at -10.2% but defense adjusted (DVOA) pass defense was 2nd at -11.6%.
The folks making this claim are forgetting that *all* defenses do significantly worse vs elite QB's. The Pats pass defense last year actually didn't fall off as much vs elite opponents as most teams do.
As an overall defense, they were 3rd in overall defense DVOA, and first by weighted DVOA, meaning on a per play basis they were playing the best defensive football in the league at the end of the season.
A lot of folks get hung up on legacy stats like PPG (points/game) and fail to realize that the Pats special teams and offense gave up far more points than average last year and the defense scored far more points than average. PPG blames the defense for all those points and doesn't credit the defense with being one of the very best defenses at scoring in the league. Adjust for those factors (as I have in a prior post) and the defense was 1st or 2nd in PPG.