Speaking as a Patriots fan, this argument is really simple; currently, Peyton Manning has more strengths and fewer weaknesses than any quarterback in the league, easily. Manning has always been stronger, more durable than Tom--he's like a cyborg. But early in his career he could get rattled, frustrated, irritated...he lacked or seemed to lack mental toughness. Tom has always had that part--he is one of the rare players who just has IT--but to Manning's credit, he has worked on that aspect of his game, and he's as tough as anybody now. Just look at what he has been able to do against us, and against other teams.
I use the 2006 AFC Championship game as a marker for this. Once upon a time, he throws that killer pick-six to Asante, he's done, finis. Colts would never come back in 03, 04. But look what he does in '06--he admits it was on him, and comes out of the locker room on fire. People often say that the pats choked away the lead, but I think it was just a tribute to Manning's new confidence. NE goes up 21 points, and then Indy scores 3 TDs, and it's a game again.
It has to also be said, that the Colts have managed to improve their defense and their OL over the years, and that really helps any QB.
And I would think that, all rivalry aside, most Colts fans realize that Brady is a special player, too.