QBs can play very well into their late 30s, and there is really nothing about Brady's skillset that will deteriorate too extraordinarily with age. If he stays healthy and wants to do it, he'll be a highly effective QB at least until he's 40 (Favre is playing pretty well at 39. The mistakes that he's making, at least, are the same mistakes that he made when he was 30, so it's not an age thing). Arm strength, accuracy, pocket presence, and ability to read coverages don't go away with age. Speed does, but he's already slower than McDaniels, so whatever.
So, in short, I think we need to keep Brady, and I don't even really see a debate about it. Cassel will be a good QB, but there are still way too many unknowns about him right now. Also, one variable that most people seem to be leaving out is O'Connell. To have drafted him in the third round, Belichick/Pioli must have seen something pretty special in him, and probably think that his long-term prospects are at least as good as Cassel's (probably better). Add in that he's even younger than Cassel, and I really doubt that where Brady will be 4-5 years from now will even enter into the Cassel discussion.
What I'm curious to see is what will happen in a few more years, if O'Connell never gets the shot that Cassel did this year. Will some QB-needy team offer up draft picks for him, solely based on the fact that he was drafted and coached up by the same people that were responsible for Brady and Cassel?