Dr. Z on Passer Ratings

[unoriginal]

I've been reading Dr. Z since I was a kid. He was probably the most famous and influential football writer in America. I still enjoy his annual All-pro columns. But he is committing the most grievous of all sportswriter sins: He has become boring. He's now Dr. Zzzzzzzzz.

If the current system is broke, I know one thing: Dr. Z's ideas to repair it are almost certainly wrong.

Apparently his fix-it idea is to re-zero the scale so that the league average leads to an average passer rating.

That's not too bad a start. They should do that after every season.

 

[solman]

I already knew passer rating was a crappy stat. I didn't know it was based on 1973 standards. That's ridiculous.

It is useful for the extremes. If a guy has a rating >100 you know he had a good game, while a rating <50 tells you he had a crappy game.

It's of little usefulness when comparing similar QBs or when the rating falls somewhere in the 70-90 range.

Really? In the last five seasons, was there ever a season in which a QB (with at least 200 attempts) with a rating 70-75 was did a better job passing that a QB with a rating 85-90?

Give me just one example.

In 2006, the 85-90 range included:

Tom Brady
Mark Brunell

The 70-75 range included:

Alex Smith
Matt Leinart
Rex Grossman
Brett Favre
Charlie Frye
Brad Johnson

A 10 point differential in passer rating maintained over the course of a season is fairly conclusive proof that the higher rated passer is the better passer.

 

[DarrylS]

Why write a negative article about passer ratings now I wonder?

Must have had lunch with Greg Easterbrook lately and the discussion had to have been about the "evil" Patriots... all a bunch of ratings driven lemmings.

 

[sanvara]

Check out this example (which I al...tive way of determining who performed better.

 

[godef]

Yeah...QB A could have thrown 12/15 for the same yardage and TDs and still ended up with the lower rating, though. PR is probably better long-term than single game, just like most stats, but it doesn't erase the arbitrary and outdated "caps" and (in my opinion) extreme overweighting of INTs.

Anyway...another baffling thing is the cap of 158.3, when they literally could have picked any other number as the max.

I think because this is actually a combination of multiple "caps" added up...

But yeah, the thing I always wondered... I believe Brady had a "perfect" 158.3 rating this past Sunday; so what if he had completed any of those 4 incompletions? or if he had thrown for additional TDs? The rating remains fixed at 158.3?

Certainly, those extra completions or TDs would impact his over-all season rating, so yes, apparently the QB rating thing is not designed for single-game comparisons.

 

[primetime]

The football outsider's DPAR ratings are better to judge a QB's performance, anyhow...

http://footballoutsiders.com/stats/qb.php

Look at how far Brady is from Manning, and how far Manning (and Garcia) are from the rest of the pack. Unbelievable.

 

[Oswlek]

The football outsider's DPAR ratings are better to judge a QB's performance, anyhow...

http://footballoutsiders.com/stats/qb.php

Look at how far Brady is from Manning, and how far Manning (and Garcia) are from the rest of the pack. Unbelievable.

In fairness to Manning, the DPAR stat takes total plays into consideration and Brady gets extra points for not having his bye yet. That said, he still has a sizable edge in DVOA which is the per play number.

Totally unrelated, are we seeing a jump in passer comp % this year, or have I just not paid that much attention to it in the early parts of the year before? Running through that list, it is surprising just how many guys are mid 60s or better.

 

[FlyingElvis75]

You can pick out weird unrealistic examples like this (no one completes 46 of 47 passes) , but in the end, over the course of an entire season, the higher rating is meaningful and is an effective way of determining who performed better.

I wrote in a follow-up post that changing the 46/47 to 12/15 would produce similar results. The point is that a record-breaking game (not entirely unrealistic except for the # of attempts, which I emphasize is not required) could result in a lower rating than a mediocre game.

I don't think anyone debates that it's "meaningful," at least over the course of a season, it's just that it's getting outdated. It doesn't reward better performances with higher ratings due to the caps, because nobody imagined that performances could be consistently that good when it was created. And it overpenalizes heavily for interceptions.

 

[FlyingElvis75]

I think because this is actually a combination of multiple "caps" added up...

Yes, it's essentially four variables, each multiplied by a coefficient and "capped," then added up. Which is why literally any number could be set as "perfect" - just change the values of the (arbitrary) coefficients or caps.

But yeah, the thing I always wondered... I believe Brady had a "perfect" 158.3 rating this past Sunday; so what if he had completed any of those 4 incompletions? or if he had thrown for additional TDs? The rating remains fixed at 158.3?

His rating for the game would be stuck at 158.3 even if he had completed all his passes. But you're right, from the standpoint of season totals, it would have a positive impact (assuming his season is less than 77.5% completion and 12.5 YPA and ~0.12 TD PA).

 

[GuyClinch]

The problem lies with the broadcasters. They need to start using more advanced statistics to talk about the game. If you could get one guy to use someting like DVOA and popularize that this problem would disappear.


Pete

 

[FlyingElvis75]

I'd say that's silly, since wins are dependent on far, far more than passing performance. In any case, PR isn't meant to assess team winning ability - it's meant to assess passing performance.


The question was - how well do wins correlate with passer rating. Not a silly question at all. To put it differently, how much of a team's success (winning percenataage) is explained by passer rating?

"Explained by" (in stats language) refers to causation, not correlation - i.e., it implies directionality. Correlation is only the strength of association between variables; it doesn't establish directionality. Classic example is ice cream consumption and violent crime - there's a strong correlation between the two. There's also supposedly a strong correlation between kneel-downs and wins.

I'm pretty certain there is a correlation between passer rating and winning percentage, but that doesn't mean that one necessarily causes the other, or even that there is a "true" correlation when controlling for other factors.

I would even see that as a weakness. If you were the greatest passer in the world, but your team couldn't win (e.g., your defense sucks), would that make you any worse of a passer?

 

[jaytyme]

Some addition passer rating issues:

* A QB that takes a sack rather than throw it away is treated favorably.

* Rushing is totally excluded (passer ratings)

* TDs are given far too much weight. If two QBs have equal supporting parts, and both pass for 18/24 for 240 yards on non-TD passes, but one team decides to run in their two one yard TDs and the other decides to pass, the one who ends up with the TDs is rated 25 points higher.

The sack/throwaway one is the one that bothers me the most.

I agree with all these points and would like to add 1 more:

* Interceptions should be weighted on a sliding scale with the 2 extremes being

- interception on 3rd and 15 thats 35 yards downfield (least harmful)
- interception on red zone 1st and 10 thats returned for a TD (most harmful)

 

[FlyingElvis75]

But over the course of a season (which is the period of time that passer rating was designed for) passer rating is an extremely effective gage of which passers performed better.

I think that's a reasonable view and usage of PR. The problem is that most viewers and media don't limit their usage that way. Hell, I even saw a game where they compared a single QB's rating based on "distance of attempt," meaning they showed a graphic with his PR for attempts of 10 yds, 20 yds, 50 yds, etc.

The key inadequacies (adjusting the other players on your team) can't be handled effectively using statistics (and are widely accepted with other statistics and in other sports).

At a minimum, I don't think that a substantially more accurate statistical measure can be created without making it substantially more complicated.

See, I don't see that as a key inadequacy; as you say, the rest-of-your-team factor applies to pretty much any stat. It's the subjective "caps" that are the problem for me. The idea that a guy who throws for 16 YPA with 1 TD per 5-6 throws is no different from the guy who throws 12.5 YPA with 1 TD per 8-9 throws, all else equal, doesn't seem right, nor does the idea that no amount of TDs can compensate for even one INT.

 

[solman]

I wrote in a follow-up post that changing the 46/47 to 12/15 would produce similar results. The point is that a record-breaking game (not entirely unrealistic except for the # of attempts, which I emphasize is not required) could result in a lower rating than a mediocre game.

I don't think anyone debates that it's "meaningful," at least over the course of a season, it's just that it's getting outdated. It doesn't reward better performances with higher ratings due to the caps, because nobody imagined that performances could be consistently that good when it was created. And it overpenalizes heavily for interceptions.

1. Its not designed to measure per-game statistics, so any inadequacies in the regard can be ignored.

2. I disagree that it over penalizes interceptions. It over penalizes interceptions on 4th and 20 from your ten yard line with 20 seconds to play. It under penalizes interceptions on first and goal from your opponents five yard line just before the end of the first quarter. If minimizing complexity is a goal (none of us could replicate the football outsiders stats in any reasonable amount of time) then you simply have to accept this situational problem.

 

[solman]

It's the subjective "caps" that are the problem for me. The idea that a guy who throws for 16 YPA with 1 TD per 5-6 throws is no different from the guy who throws 12.5 YPA with 1 TD per 8-9 throws, all else equal, doesn't seem right, nor does the idea that no amount of TDs can compensate for even one INT.

To my knowledge, no starting QB has ever exceeded ANY of the caps over the course of a season.

Even with the amazing season that Brady is having, is is still not hitting any of the caps (although one more TD would do it).

If you are concerned about per game numbers, it would be trivial to remove the caps, but just how meaningful is a 200 vs 180 passer rating comparison.

Without caps a 1 TD/1 attempt game by LdT would score over 300 points from the TD per attempt component alone. This is utterly meaningless to me. What makes it meaningless is the fact that he had just one attempt.

 

[FlyingElvis75]

1. Its not designed to measure per-game statistics, so any inadequacies in the regard can be ignored.

The fact that it is used to measure single-game statistics is evidence that it can't be ignored.

2. I disagree that it over penalizes interceptions. It over penalizes interceptions on 4th and 20 from your ten yard line with 20 seconds to play. It under penalizes interceptions on first and goal from your opponents five yard line just before the end of the first quarter. If minimizing complexity is a goal (none of us could replicate the football outsiders stats in any reasonable amount of time) then you simply have to accept this situational problem.

Again, I posted this earlier in the thread: In the worst case (perhaps the first situation you mentioned, 4th-and-20 from your 10 with 20 seconds left), you lose a touchdown you might have scored and allow the opposition to score a touchdown. That's a two-TD swing. From a scoring perspective, at the very worst, two TD passes wipe that out. Equating 1 INT with -2 TDs is as justifiable as any blanket assumption made about any major stat (and yes, I'd agree that if situational stats are your concern, football outsiders is the place to go). The idea that NO amount of touchdowns can overcome even one interception is not objectively justifiable, in my view.

 

[FlyingElvis75]

To my knowledge, no starting QB has ever exceeded ANY of the caps over the course of a season.

Even with the amazing season that Brady is having, is is still not hitting any of the caps (although one more TD would do it).

If you are concerned about per game numbers, it would be trivial to remove the caps, but just how meaningful is a 200 vs 180 passer rating comparison.

All other major stats are uncapped; I'm not sure why that would be a problem?

Without caps a 1 TD/1 attempt game by LdT would score over 300 points from the TD per attempt component alone. This is utterly meaningless to me. What makes it meaningless is the fact that he had just one attempt.

The NFL already requires an average of 14 attempts per game to "qualify."

 

[solman]

The NFL already requires an average of 14 attempts per game to "qualify."

And that is sufficient to remove the problems that you are complaining about.

The same can be seen in rushing stats where some RBs have 10 yards per carry after one game.

After a few games with a minimum number of carries per game, the long runs get averaged out.

 

[FlyingElvis75]

And that is sufficient to remove the problems that you are complaining about.

The same can be seen in rushing stats where some RBs have 10 yards per carry after one game.

After a few games with a minimum number of carries per game, the long runs get averaged out.

No, it isn't.

Jake Delhomme, Jeff Garcia, or Peyton Manning - which do you feel is the best passer?

 

[BradyManny]

I question the timing, but agree with the content.

Ditto. But for so long we've heard the BS that Brady isn't as good statistically as Manning (or other legendary QBs). After hearing that for so long and now that he's on pace to demolish every single season passing record, I'm not about to start saying the stats are flawed.