It's like saying that the only stat in baseball that truly matters is runs scored.
It sounds good, but it's not really true.
To make this an absolutely absurd example, imagine a team dominating time of possession, taking forever to move the ball down field, limiting your opportunities to possess the ball so much that you only get 4 possessions - 2 in each half. And let's say the other team manages to score 17 points in those long four possessions. You look at the end of the day and say, wow, great, our team only allowed 17 points - they must have done a good job!
But of course, they didn't. They allowed 2 TD and a FG in just four enemy possessions. And your own offense, as good as it may be, only comes up with 14 points (2 td in 4 possessions), and you lose 17-14.
Obviously an extreme illustration, but it captures the point well enough - there's more to a good defense than simply reducing the number of points the other team scores. Bad defense also prevents your own offense from having more opportunities to score, and that may very well matter.
I think the second SB against the Giants was a less extreme example of this. The Pats' offense wasn't great, but the defense "only" allowed 21 points, but the Patriots only got 9 possessions, as the Giants kept the ball for 15 minutes (a full quarter) more than the Patriots, thus dominating the flow of the game. And yes, that mattered in the end.