Because the Pats won on defense, not offense.
You agree the latter part of the dynasty was more Brady and I think most people would agree with that (anyone who doesn't can chime in here). I think most people would agree too the
more about Brady started in 2007.
So let's look at some numbers and details from 2000 to 2004 (which seem to be the seasons mostly in question)...
2000 (5-11)
Offense P/G: 25th
Defense P/G: 17th
Offense Y/G: 22nd
Defense Y/G: 20th
Bledsoe: 5-11, 77.3 passer rating
2001 (11-5)
Offense P/G: 6th
Defense P/G: 6th
Offense Y/G: 19th
Defense Y/G: 24th
Overall team was looking pretty bad through Week 2, having gone 5-13 with Belichick & Bledsoe. The offense would improve significantly with Brady as the starter and so did the team's fortunes.
Brady: 14-3, 86.5 passer rating (up 9.2 from Bledsoe in 2000), Pro Bowl, SB36 MVP
Season Highlights: 364 and 2 TD's in OT win vs SD; 80% and 3 TD's in win at Indy; 75% and 4 TD's in win vs NO; 138 in 4th & OT in Snow Bowl; game-winning drive in SB36
2002 (9-7)
Week 16 loss at home vs Jets led to the only non-division winning season of the Brady era.
Season Highlights (Brady): 294 and 3 TD's in opener vs Pittsburg; 410 and 4 TD's in shootout win over KC; led NFL in TD's
2003 (14-2)
Offense P/G: 12th
Defense P/G: 1st
Offense Y/G: 17th
Defense Y/G: 7th
Brady: 0 TD & 4 INT's in opener put him in a huge hole statistically; in 11 of remaining 15 games, all wins, he had 14 TD's & 0 INT's; SB38 MVP
Season Highlights (too many to mention): 82-yard bomb to Brown in OT win at Miami (still an all-time favorite replay); 350 and 3 TD's including game-winning pass to Givens with 0:30 left at Denver; 368 in OT win at Houston where he connected with Graham with 0:40 left in regulation before leading a game-winning drive in OT; 4 TD's in Week 17 rematch vs Buffalo; Divisional win over Tennessee in literally 20-below; 354, 3 TD's and game-winning drive in SB38
2004 (14-2)
Offense P/G: 4th
Defense P/G: 2nd
Offense Y/G: 7th
Defense Y/G: 9th
Brady: Pro Bowl; highest postseason passer rating of career (109.4); only QB to win back-to-back Super Bowls in 2000's
Season Highlights: 335 and 3 TD's in opener vs Indy; 300 and 2 TD's at Buffalo; 5 TD's, 0 INT's & 109.4 passer rating in postseason
Brady looks like a definite elite quarterback 2001-2004. How can you even question it?