Yeah, I agree with you. Best way to beat NE (or really any good team) is a good pash rush on defense and a good offense capable of scoring multiple ways. Jacksonville may have a better overall defense than Pittsburgh but that does not make up for their offense being worse. BB/Josh will always find a matchup to exploit against your defense, and Brady is capable of making plays at times that simply cannot be defended - you need to be diverse and talented on offense to keep up. If your offense is basically "we can run good and sometimes our QB throws a good pass" (i.e. Buffalo's offense) then it's not a big struggle for the Pats defensively.
I get "any given Sunday" but when it comes down to it, it's about the offenses. You can have an insane defense, but Brady still did just fine against Seattle in SB49 and that was a hell of a defense.