Q. Is it subjective?
A. There is a perception, mainly from people who watched the QBR special, that QBR has a lot of subjective components. Some who watched the show thought that the expected points were allocated subjectively by people watching games -- give this guy a point or take away a point. This is not true and something we need to clear up.
What underlies QBR is expected points and win probability. In determining expected points and win probability as they relate to field position, down, and distance, there is no subjectivity other than slight differences in how these models are built (which is why AdvancedNFLstats.com doesn't have exactly the same numbers we have). These differences are definitely small.
The part of QBR that could be cynically called "subjective" is that there are judgment calls with regard to what are dropped passes vs overthrows or underthrows or defended passes. ESPN's video trackers have strict guidelines on how to chart these items so that they are consistent across the different people doing charting. If you as a fan go out and chart these yourself for a game or two, you will see how several calls are easy, but some are quite hard to judge. We have standards that make things more uniform and every game is done twice to reconcile inconsistencies. Despite the standards, the gray areas will still exist and, because they exist, the division of credit quantitative analysis described below is important. That analysis is what says that a "drop" isn't necessarily all about a receiver because there are gray areas in drops.
Notably, the kind of judgment calls here are not unique. Every week, statistics like hurries, tackles, or targets get used but have similar judgment necessary to decide them. Neither of these are official NFL statistics and both come with clear gray area. Coaches are known to spend hours going back to evaluate credit on various plays. Our hope is that any statistics used to evaluate individuals in football come with analysis to help split the credit in these more gray areas. We did that analysis to limit subjectivity.
