Exactly.
As far as what should be challengeable? That's a tough one. IMHO any thrown flag should be challengeable, period. I'd argue PI, for example, is more often than not going to be decided more justly from video replay than in real time on the field. However, it gets dicey if you can challenge anything that didn't draw a flag. I'd bet if you scanned the field looking at every individual battle, you could find a penalty on just about every play (at least by the letter of the law). That in no way would be a good thing for the game.
This plan won't change the bad non penalty call situations but it would likely have a good effect on bad penalty call situations.
I don't see why you can't have your (non-calling) cake and eat it too if you base your challenging criteria on a)the certainty a penalty was/was not committed, and b)the call/non-call in question has a
direct and obvious impact on the outcome of a play.
Anything that's called a penalty, by default, has a direct and obvious impact on the play's outcome, so all penalties, as mentioned, could be reviewed.
For a non-call, your window of operation tightens, but this way simultaneously enhances getting the calls that matter right, while mitigating abuse of the new system.
Let's look at two examples from the Houston game:
1) Julian Edelman is clearly and demonstrably hit after signalling a fair-catch. This can and should be challenged, and the ruling of a non-call would be overturned.
2) Danny Amendola (I think) draws an illegal hands to the face penalty. Had this NOT been called, I don't think it could be challenged. While it is clearly a penalty, outside of Amendola *possibly* being Brady's first target, the penalty had no direct or obvious impact on the play. If Brady *had* targeted Amendola, then the impact becomes direct and the non-call should be challenged.
Also, obviously no 'blanket' challenges should be possible (e.g. offensive holding, somewhere, at some point). Challenges would have to be player-specific. Even penalties such as offsides would require you to pick one player you believe was guilty/innocent (and if you think the O drew your player offsides, challenge false start on the offending player).
Count me in the lot who don't get nostalgic for reffing 'tradition'. The more objectivity, equality, and clarity you can instill into the game, the better. The argument that the games would be longer is blatantly ridiculous; we are not advocating an increase in challenges, but rather more freedom in WHAT we wish to challenge.
If PI is called on a tick-tacky call but where there was contact, no coach is going to challenge that, because there's no way it would be overturned. That grey area, due to the complexity of the game, is intrinsic, and I have no immediate solutions to how you remove it.
But calls heard around the world, that are obvious, wrong, and alter the outcome of games need to be stopped. At the VERY least the coaches need to have the power to question them.
Right now we're not even entitled to an explanation.