He was 3-2 vs New England. 11-11 vs every other team. And neither of the teams that lost to NE (2003, 2004) were among his best.
You make good points. In particular with Manning going 11-11 against other competition.
My main thing to add would be that the Patriots' constant success caused Manning and other AFC teams to be perpetually a lower seed on average than would have been the case against any other NFL squad over that period. That fact matters when one considers that home-field advantage was so important to Manning and Brady head-to-head in the postseason. When Manning was with the Colts, the three playoff games between Manning/Brady were decided by home turf; likewise, when he went to the Broncos, the two playoff games between the pair also aligned with the home team.
If the Patriots were an average squad, the Colts (who were also perennial contenders) would be in a position to be a much higher seed. With the Broncos, who managed to be #1 seed three times with Manning despite the Patriots being in the same conference, the Broncos advanced twice to the Super Bowl and won once.
One can only imagine, based on the importance of home-field between these two, what would have happened in 2004 if the regular season match up we won was at Indy instead of New England. Say Indy wins that game and finished tied at 13-3 with the Patriots along with the tiebreaker for #1 seed. Based on these trends, Indy then wins in the championship game hosted in Indy instead of Foxboro and advances to the Super Bowl. I'd like to think the Patriots would not have been beaten twice in the same year, but it is an intriguing scenario.
None of this makes Manning less of a choke artist in a number of important games. But I do suspect if you place the Colts/Broncos in the NFC during that window, Manning probably advances to 1-2 more Super Bowls and probably retires with 3. But who knows, it is purely hypothetical.