Of course if his play slips to the point where playing him is detrimental to the team, then it will become an issue. However I don't see that happening before 2017.
Don't forget that Brady won his job because he was a better decision maker than the much more physically talented Drew Bledsoe in this offense that requires and rewards great decision making skills. If anything Brady's decision making is even better with well over a decade of experience. So while his physical skills slowly erode, his decision making ability should allow him to get to 40 and STILL be an adequate starter on a championship contending team.
However in 2017 a combination of a new contract and what to do with Garapolo will necessitate a hbig/hard decision. AT THAT time, all the crap we heard last week WILL become legitimate fodder for discussion.