The 3 factors that determine the passer rating are yards per attempt, TD's per attempt, and INT's per attempt.
Manning beat him out on two of those categories. Of all the QB's to throw for over 40 TD's though, Brady is the only one to play his home games outdoors in a cold weather climate. Marino played in Miami and Warner & Manning both played in domes.
It makes a big difference. Consider the fact that before the weather turned, Brady's rating was around 139.
Putting Brady in a dome would be like putting Babe Ruth or a juiced up Barry Bonds in Coors field.