I guess I don't see time as you do.
If Rivers was here, our past wouldn't be replicated, so I can't "give" Brady's victories to Rivers, cause they don't exist.
So, to formulate my train of thought: if we had all the same pieces, but with Rivers in place of Brady, we would still be one of the best teams this year, we would still have been a dominant team for more than a decade (he entered the league in 2004). And, in one of those years, I believe that Rivers would win a Super Bowl (or 2), helped by BB and all the players that were here, because he's a good QB.
I don't think the Patriots would've won 5 Super Bowl and been to 7, though. And we might have gone through some years without the AFC east crown