I think the 2004 team takes it by a wide margin. Forget about the best team in franchise history, the 2004 team deserves recognition as perhaps the best team of all time. Consider the following.
1) Depth - When I evaluate this team, I look at the starting roster that began the year and established the NFL's longest "true" winning streak at 21 games. I look at the depth of the roster and how they were able overcome all those injuries to post an unbelievable 14-2 season. The number of injuries was ridiculous, especially to the secondary where both starting corners were lost for the season. And then we even lost our #3 (Samuel) for a few games on top of that. It was so bad that this was the year where Troy Brown was forced to play DB.
2) Style - The 2004 team played a superior brand of football. They were too physical for the finesse teams and too skilled for the physical ones. They were one of the few teams in NFL history that achieved complete balance across the board. They could run, pass, and defend against both and their special teams were top notch as well. Another advantage that this team had, was that it was so difficult to stop their passing game b/c they weren't dependent on just one or two guys. Between Branch, Givens, Brown, Patten, Faulk, and Graham, Brady was able to spread the ball around to such an extent that it was completely ineffective for defenses to key in on any any single target. Mix in our dominant running game and defenses had no chance. They might get lucky here or there but there was never a solution to completely stop us.
3) Coaching - This was the last year that Belichick, Crennel, Weis were together. That combination of coaches filling their respective roles gave us a tremendous advantage against every team in the league.
4) Intangibles - Toughness, resiliency, and an ultra competitive desire to win. This team had an abundance of all of those things. The image of Bruschi ripping the ball out of the hands of the Colts RB during the playoffs describes this team better than I ever could. No matter how many bad breaks this team was dealt, they refused to lose. The just kept fighting and fighting. Watching them was like watching Rocky take down the Russian.
5) Accomplishments - Just check out their resume. It speaks for itself. The 2004 team was an extension of the 2003 team only with Corey Dillon running the ball instead of Antowain Smith. They had back-to-back 17-2 Super Bowl Champion seasons with an astonishing 21 straight wins bridging the two seasons. Those 34 wins over two seasons are an NFL record. And they did it all while playing a difficult schedule as well. These numbers weren't padded by a shoddy schedule. Quite the contrary. I don't remember the specific record but it they were something like 22-1 vs winning teams or perhaps it was playoff teams. Amazing!