Have you watched the America's Game episode on the '01 Pats? In it, they show Brady's draft weigh in. He's shirtless. Thats right, shirtless Tom Brady
!!!! He looks like an anorexic teenager. The physical knock on him was that he was simply NOT athletic (ran a 5+ 40 time and weighted I think closer to 200 than 210). He was a skinny, scrawny, slow guy. He had the height but not the big body. I remember him throwing plenty of long balls and having a good arm (albeit nothing super spectacular) while playing at Michigan but yet he didn't have the type of "laser rocket arm" reputation of Manning or Leaf.
He was overshadowed because Michigan's offense ran primarily through Anthony Thomas and the running game so a lot of people perceived him to be a "system qb". He also had some good college receivers (including first round bust David Terrell) and some people gave the receivers too much credit and not enough to Brady. I think those are the main reasons. In general, QB is probably the hardest position to translate from college to pro. Scouts just weren't sure.
As happens too often in the draft, people focus too much on the physical and not enough on the intangibles. The guy was smart (he studied Organizational Studies which at Michigan is actually a pretty elite program, and one of my econ professors at UofM actually taught TB and said he was a very good student), he was a great leader, a senior captain, a hard worker who fought off the most hyped QB to come into Michigan (who abandoned the team to play baseball later on), completed 62+% for 2,600+ yds, 20 TDs, 6 INTs senior year, and he led some pretty epic comebacks as well as some epic near-comebacks, and was at his best in pressure situations.
Hope that helps.