I've noticed this a lot too when going to games. The crowd doesn't get loud when the other team is in the huddle, but only when they get to the line and then not always loud enough in enough time to really affect them.
I've heard the stadium get loud and it's usually when the Pats get touchdowns or some huge play is made. The loudest I've heard the stadium (and I've said this many times) was the dropped pass that ended the Ravens comeback chances in that 2009 regular season game. Deafening...
Of note, during the 2011-2012 years, when the D could never get off the field on third down, I noticed a lot of people just got tired from consecutive drives of screaming their heads off, only to have the opponent convert on the third down and then the cycle starts again, over and over.