Re: Pro Football Focus is pretty funny.
I'm all in favor of ice-cream-eating nerds trying to devise some sort of home-brewed analytic system to try to contribute something to our understanding of football that we're not getting from the storyline and superstar obsessed mainstream sports media and their continued adherence to woefully inadequate traditional counting stats.
When there's a conventional wisdom as calcified and reflexive and resistant to new information as is wont to happen in professional sports, there's always room for intelligent people with fresh perspectives and impartial agendas to bring something new and exciting to the table. In the case of baseball, a game that lends itself to statistical analysis like no other, the nerds and their "sabrmetrics" have revolutionized the game to the degree that there is no longer a single team in the league whose front office doesn't employ some of their methods.
Football, more than any sport, relies on players executing a complex 11-man choreography on every play, with every individual player's success dependent on internecine layers of mutual contingency, that is strongly resists quantitative analysis of players in isolation. Accepting, then, the limitations of statistical analysis to discussion player-units working in concert, as opposed to individual players, there remains a lot that a new statistical approach can tell us that we can use to augment our understanding of the game. For this to be effective, the approach must be rigorous and exploratory as opposed to didactic, always remembering that the data exists independently of our interpretation of it, and that we must use our understanding of football to guide how we look at the data just as much as we allow the data to shape our understanding of football.
My personal favorite NFL site, FootballOutsiders.com, accomplishes this quite well. The write-ups of their statistical analysis are always just as much about understanding the limitations we need to impose in interpreting their data as it is about using their data to comment on the reality of the NFL. If the results are counterintuitive, Aaron Schatz, the site's founder, will work diligently to understand and explain what's behind the result, and to qualify what and how much we should take from it. Basically, he approaches his data with the rigor and caution of a scientist, trying to find a way to reconcile the data with our current model of understanding as opposed to blindly supplanting our current understanding with the implications of his data.
Profootballfocus does pretty much the opposite. Their write-ups are full of bombast and posturing, pitched at brow-beating you into accepting their assertions based on the strength their proprietary data, which they won't even show you unless you pay them. They don't seem particularly interested in "showing the work" behind their claims, most likely because they know it won't stand up to scrutiny. Many have already pointed out some inherent flaws in their methods -- trying to break down plays with only the tv footage and no knowledge of what was *supposed* to happen on a given play, using volunteer data collectors with no quality control, etc. -- but there's little point engaging with them that much when they're still so opaque about the rest of their methods. It's hard to take them seriously as someone who could really contribute to the field of football analytics when they keep insisting that we pay no attention to the man behind the curtain.
Anyway, sorry to go on about this at such length. I just wanted to argue against anyone dismissing the legitimacy of the entire field of inquiry based on the work of hacks like these guys.
