Brady was a disappointment at TE last season, so I don't see how you call that a downgrade. We don't know if ILB or CB are better, worse or a wash. The OLB position is better this year than last year, not worse. As for the OL backups, once Neal returns that argument disappears. As for the "always want to see....", again, that's just unreasonable.
Break it down.
QB - Wash to slight improvement, depending upon O'Connell's development
RB - Improvement with Jordan over Eckel
TE - Wash to better, depending upon Thomas
WR - Downgrade to upgrade, depending upon Jackson's progress
OL - Post Neal return, it should be a wash, prior to that, a downgrade
DL - improvement due to Seymour's health
LB - Unknown. More speed, less experience. If Seau returns, upgrade
S - Should be a definite, and significant, upgrade
CB - Likely a pretty big drop for the starters at the beginning of the season, with that gap likely closing over time. The nickel is unknown, but Gay struggled last year, and it shouldn't take much to improve upon his performance.
P/K - Wash