It takes more than just being physically talented to be on the patriots, they require a level of intelligence to understand their system as well as the willingness to put in the work to learn it. I don't think you have to be a savant, but you have to be willing to take time to study, and practice. There is a limited number of players who can fill these roles, so yes in a way Brady, Gronk, edelman, Moss, Welker amongst the others are few and far between.
There was an article, which I can no longer remember or find that talked about this and how the Patriots offensive plays are unique amongst the league. Most teams have a set of plays they will use each game, and may make adjustments to them depending on the opponent but they stick to those plays every game. The Patriots custom tailor every game to their opponent, increasing the number of plays a player needs to learn. In addition, each play can have several things happen depending on the look of the defense after the ball is snapped, so the running backs, line men, receivers, etc all have to be on the same page and adjust accordingly.
The article compared the Patriots way to a team like the Colts who have a base set of plays they always use, and their goal is to execute those plays perfectly, so thats all they ever practice. They depend on defenses making errors or mistakes. The patriots show a new look each time and are unpredictable. It also went on to say this is partly why Ochocinco did not make it with the Patriots.
As I recall the article used the analogy of a rented tuxedo which is made for a broad set of people, versus a custom tailored Tuxedo that is made specifically for one individual. The patriots make a new tux for each opponent.
Bottom line, Colts and many other NFL teams systems are fairly easy to learn to an experienced player, where as Patriots system is hard for just about everyone unless they are putting in the time.