Look, I'm as big a Brady fan as you'll find- just look at my username. Watching the Pats without Brady just isn't going to feel like watching the Pats, and this is the first time that I've felt that way. But the Pats are still our team and Matt Cassel is now our QB. Acknowledging that, moving on, and pulling for him for the rest of the year doesn't mean that you're turning your back on Brady- in fact, I'm pretty sure that that's what he's doing himself and what he would want you to be doing anyways. Hope for the best for the guys who are there.
Absolutely nobody is saying that the Pats won't miss Brady in a huge way. They will, both because he's their emotional and tactical leader and because they're simply not nearly as talented an offense without him as they are with him. Anyone who says we'll be "fine" basically means that the Pats with Brady were so good (this team did go 16-0 last year) that even without Brady they should still be good enough for 10-12 wins and a playoff berth, and from there... well, weirder things have happened than the Pats getting hot at the right time. Just look at last year's SB- once you're in the playoffs you don't have to be the best team to win it all, and you don't have to have the best QB.
Nobody's saying that Brady isn't worth his salary, or isn't the best QB in the NFL. Without him, the team has gone from overwhelming favorite to maybe a contender. We're just saying that the rest of the team is good enough that, even if the Pats aren't nearly as good without Brady as they are with him, they still have an outside shot at surprising a lot of people and making a run.