Our Coach and Economics (for all Mathematics Enthusiasts)

[holyredeemer]

I think you're going WAY too far in the other direction. What you've said renders play selection meaningless -- might as well run it up the middle on 3rd and 15 if all that matters in this game is reaction speed, and execution.

Absolutely every NFL coach relies on "averages" to make decisions. That's why they're passing on 3rd and long, punting rather than attempting 70-yard field goals, etc. What's unusual about Belichick is his willingness to reconsider longtime conventional football wisdom when data point him in a different direction.

The studies described in that article do NOT tell you what the right move is in a particular real-world game situation, given the myriad variables of personnel, field, etc. They DO suggest that coaches should adjust how they make those decisions to avoid common biases in judgment. E.g., the fact that overall, NFL teams punt and go for 2-point conversions too often shows that the current conventional wisdom is flawed. A competitive advantage can be gained by fixing those flaws, and Adams & Belichick will be out in front trying to grab that advantage.

Probably the clearest example of this is in the part about the draft and "hyperbolic discounting." We all know that the Pats trade picks into the future to improve positioning more than any team we've ever seen. They're systematically taking advantage of the established "1 round up" for a future pick -- which for teams with stable management is simply an irrationality in the market. It's basic draft pick arbitrage.

By reaction speed and execution, I wasn't just referring to on the field play. I'm at fault for not explaining what I meant better. I was referring to all aspects of the game. Whether it be on the field, to the film room and to the coaches preparing and calling the plays. Everything you need to know is right in front of you on the video tape. That's all I was trying to get at. A lot of what has been said is interesting, but in the bigger scheme of things, there is far too much involved to take any of the information presented about a team into account.

I'm not disagreeing with anything you have said. Your post was very well presented. But, I just take all of the statistical information with a grain of salt when it comes to what is successful and what is not.

 

[patfanken]

This is a great thread. Thanks guys for all the links. It was fascinating reading. The posts were great too. Let me tell you, if I were back coaching again, I'd take a much more aggressive approach to 4th down.

 

[makewayhomer]

By reaction speed and execution, I wasn't just referring to on the field play. I'm at fault for not explaining what I meant better. I was referring to all aspects of the game. Whether it be on the field, to the film room and to the coaches preparing and calling the plays. Everything you need to know is right in front of you on the video tape. That's all I was trying to get at. A lot of what has been said is interesting, but in the bigger scheme of things, there is far too much involved to take any of the information presented about a team into account.

I'm not disagreeing with anything you have said. Your post was very well presented. But, I just take all of the statistical information with a grain of salt when it comes to what is successful and what is not.

I think you're agreeing with the original post without realizing it. The stats come from plays of course - the numbers aren't invented out of nowhere. The advanced studies like this one aggregate as large a sample of possible in order to get the most meaningful results.

A baseball example: every single ball and strike call that every umpire makes is now available in a database. It can be cross referenced against pitch location data. Aggregate the data for thousands and thousands of pitches and you can see which umps call strikes high and outside, and you can train your pitchers to throw this pitch when that ump is working. Some teams bother to do this...some don't. The data is all there for the taking, whether on film or through scouting, bit in order to make it meaningfull and impactful you sometimes need to take a huge swath of data. This doesn't tell you WHAT to do but it can sure help turn a play with a 55% chance if success into 1 with 65% chance and that is a huge edge to pick up

 

[MoLewisrocks]

It's impressive that Belichick would read a paper, but he supposedly has intellectual curiosity about all sorts of non-football topics.

I saw the theory in control systems, but the bottom line still comes down to modeling the problem. It isn't just plugging in numbers. You have to identify what is relevant to determining an outcome. As that paper notes, much of the data is based on third down and is used to establish a model for fourth down planning. The data is also based on 1st quarter decisions, not 4th quarter and finally assumes the coach is risk-neutral, not risk-seeking or risk-averse. A basic understanding of football would probably tell you these are fairly major assumptions in professional football. I suspect based on these assumptions, Belichick is probably not employing the paper's theory but rather his own knowledge of teams' tendencies, his tendencies and basic probability.

If you are having trouble falling asleep, most of the theory and equations can be found here.

When RAC and Charlie left BB was asked how he would manage the impact of losing his OC and DC and his response was as long as I still have Ernie we'll be fine...

Ernie helped him fashion a defensive game plan based entirely around statistical probabilities and trends relative to a Mike Martz offense. The concepts were startingly simple. The positioning of Marshall Faulk was the key to run/pass. Martz never ran the same play out of the same set in a single game. Disrupting the timing on Warner's throws created an insurmountable issue for the Rams offense.

So the keys to the defensive game became where is Faulk lined up, forget what you think you see and hit anything that moves coming off the LOS.

I recall reading somewhere that Steve Belichick was a pioneer of sorts in formulating data on trends and probabilities in the game during his college coaching days. I believe that Ernie's fascination with Steve's book was what led to the seemingly odd friendship between the stats nerd and the jock in college, before Bill even considered persuing a career in coaching.

 

[weswelker#83]

Evaluating Risk In NFL Playcalling
There is also this nice presentation by Alamar,who recently presented a paper at the NESSIS conference in Boston regarding risk aversion and how coaches should aim for higher-variance plays in their playcalling.

That presentation is linked here.
http://www.amstat.org/chapters/boston/nessis09/presentation_material/Alamar.pdf

• Risk can now be thought of as the probability that a
play will produce negative NEP (risk factor – rf)
• Comparing run plays and pass plays demonstrates
that passing (rf = 0.57) is less risky than running
(rf=0.62) for all plays.
• Looking at specific situations, the risk profile
changes. On 1st and 10 running (rf=0.66) has a much
higher risk than passing (rf=0.53), while on 2nd and
3 running (rf=0.35) has a lower risk factor than
passing (rf=0.44).

 

[olschool]

When RAC and Charlie left BB was asked how he would manage the impact of losing his OC and DC and his response was as long as I still have Ernie we'll be fine...

Ernie helped him fashion a defensive game plan based entirely around statistical probabilities and trends relative to a Mike Martz offense. The concepts were startingly simple. The positioning of Marshall Faulk was the key to run/pass. Martz never ran the same play out of the same set in a single game. Disrupting the timing on Warner's throws created an insurmountable issue for the Rams offense.

So the keys to the defensive game became where is Faulk lined up, forget what you think you see and hit anything that moves coming off the LOS.

I recall reading somewhere that Steve Belichick was a pioneer of sorts in formulating data on trends and probabilities in the game during his college coaching days. I believe that Ernie's fascination with Steve's book was what led to the seemingly odd friendship between the stats nerd and the jock in college, before Bill even considered persuing a career in coaching.

I saw somewhere, that Ernie Adams is actually teaching a course at MIT.
Brains over brawn has always been a formula for success.

 

[patchick]

Evaluating Risk In NFL Playcalling
There is also this nice presentation by Alamar,who recently presented a paper at the NESSIS conference in Boston regarding risk aversion and how coaches should aim for higher-variance plays in their playcalling.

That presentation is linked here.
http://www.amstat.org/chapters/boston/nessis09/presentation_material/Alamar.pdf

I have to geek out on this one for a minute...

If I understand the slides correctly, this study seems weirdly simplistic. The authors evaluate options solely on expected points on that possession, with ZERO weight given to time of possession and opponents' ensuing field position. IOW, according to this model a possession that starts at the 10 yardline and consists of 3 straight incomplete passes carries the exact same value as a possession that starts at the 10 and ends with a punt 10 plays later at the 50.

I guess that's fine in a bubble, but they blithely say that their analysis "Could easily be expanded to include effects of 'next drive' or rest of game or half." If it's really so "easy," I'd say "go ahead and do it, then." Because prima facie, there are plenty of reasons to think the results might turn around. E.g., choosing to pump up variance should have the result of shortening drives, regardless of outcome. That has a significant effect on both defenses.

It's not like NFL coaches don't know that the expected value of a passing play is higher than a running play. Heck, we ALL know that. But this isn't baseball, where expected runs in the inning can be a be-all and end all-without repercussions for the rest of the game. (With a few exceptions, like swapping in relievers and pinch hitters.)

 

[Tunescribe]

BTW- the most interesting part of that article that was linked was the fact the BB responded to the guy who wrote the paper. He told him he should also consider the emotional factor of the team if they FAIL to pick it up. What was even MORE interesting was that the professor redid his numbers with that in mind.....and found out that it STILL didn't matter. Teams punt too often.

I'd think that if you went for it often on fourth down and made it most of the time it would have a demoralizing effect on the defense. Not only did they fail to stop you when it counted most, but they then know they're likely in for four more downs. If you have a good defense of your own to back you up, it would seem to make sense to go for it on fourth-and-three or less inside the opponent's 45 yard line.

 

[Wildo7]

I read this article a while ago and it makes perfect sense to me that statistical analysis favors more aggressive/higher variance decisions that go against "gut" feelings. The comparison that comes to mind is poker, where the game has evolved to the point where loose aggro players are dominating by making the better decision over a statistically significant sample of hands despite more variance in the short term results. It seems logical that coaches would wake up to the value of playing percentages to that degree, but they are generally a stubborn group that's set in their ways and slow to change. Not surprising at all that BB leads the pack in this area.

The one thing I'd like to see is how they ran the analysis of what BB calls the emotional let down of coming up short of 4th downs. I'd hazard a guess that the result the guy comes up with is a little skewed simply because currently teams that are dominating the game are more likely to go for it on 4th and short (in garbage time especially).

 

[PatsFanSince74]

Interesting. ESPN did an interview with a very successful HS HC in Arkansas (Little Rock) who virtually NEVER punts the ball. That includes time when he was in his own red zone. This is a team that has won 2 state titles in the last 7 years and is has made the playoffs every year during that time. He cited some stats that started him on this trend, and over the years he punted fewer and fewer times. Last year he said he punted only 3 times. . He also stated that he onside kicked more than half the time and had 7 different onside kick plays. Like punting he believes that the risk is worth the reward.

IIRC it was on the show that follows Mike and Mike.

BTW- the most interesting part of that article that was linked was the fact the BB responded to the guy who wrote the paper. He told him he should also consider the emotional factor of the team if they FAIL to pick it up. What was even MORE interesting was that the professor redid his numbers with that in mind.....and found out that it STILL didn't matter. Teams punt too often.

a similar point is made by Carroll et al in the book I cited above (The Hidden Game of Football). they show that it's statistically wiser to go for it on fourth and short most points on the field and certainly from inside your opponent's territory (unless, of course you can win or tie the game in the final seconds with a high percentage three points) rather than punt or kick a FG, even if its a chip shot. i've noticed that BB is doing this more often this year, BTW.

Belichick's comment is intuitively right on target, as we would expect, and is why I said in my earlier post that the models are not "robust" enough at the present time. I'd be curious to see how the guy "redid" his numbers but assume it was based on a lot of data about how teams had fared after making similar decisions.

These models really won't work in real time until the computing power is available that can instantaneously take into account every variable from down and distance, field position and time of game to the tendencies of every player likely to be on the field from both teams in similar situations, the experience of both teams in similar situations, a database of all nfl teams in similar situations, the possible impact of not making the play, the weather, the stadium and a dozen other intangibles. Considering that it would take up to 10 seconds to input down and distance, field position and time of game, that would give the computer less than five seconds to amass all the relevant data and decide who the most likely players on the field would be from both teams and then spit out two numbers (the median probability of success and some indicator of the uncertainty around that number) and give them to the HC, who would then make the risk/reward tradeoff in time to get the right players on the field.

I think this means that we have to wait for quantum computing and its promise to put a supercomputer in the equivalent of a laptop or mobile phone.

and, in the end, the very best that will be able to do will be to try to hope to mimic the brain of a guy like Belichick!

 

[PatsFanSince74]

I have to geek out on this one for a minute...

If I understand the slides correctly, this study seems weirdly simplistic. The authors evaluate options solely on expected points on that possession, with ZERO weight given to time of possession and opponents' ensuing field position. IOW, according to this model a possession that starts at the 10 yardline and consists of 3 straight incomplete passes carries the exact same value as a possession that starts at the 10 and ends with a punt 10 plays later at the 50.

I guess that's fine in a bubble, but they blithely say that their analysis "Could easily be expanded to include effects of 'next drive' or rest of game or half." If it's really so "easy," I'd say "go ahead and do it, then." Because prima facie, there are plenty of reasons to think the results might turn around. E.g., choosing to pump up variance should have the result of shortening drives, regardless of outcome. That has a significant effect on both defenses.

It's not like NFL coaches don't know that the expected value of a passing play is higher than a running play. Heck, we ALL know that. But this isn't baseball, where expected runs in the inning can be a be-all and end all-without repercussions for the rest of the game. (With a few exceptions, like swapping in relievers and pinch hitters.)

You have to read Carroll's book, The Hidden Game of Football (see my posts above). That's exactly what he tries (!) to do, i.e., take into account down and distance, field position and time of game. His problem, as I said elsewhere in this thread, is that his output is one number, representing the most likely outcome but without indicating the standard deviation or any other measure of the inherent volatility around the prediction.

And yes, you're completely right; the expected value of a passing play is higher than a running play, but the volatility around that value can be quite significant depending on the circumstances. So, saying that the most likely outcome of a pass over the middle to Wes Welker is a gain of 12.5 yards (making that up) and the most likely outcome of a pass down the field to Randy Moss is 22.7 yards doesn't take into account the standard deviations around both of those numbers. In other words, the "outputs" of 12.5 and 22.7 are simply one number along two curves that could have very different distributions.

 

[Synovia]

Perhaps if a bit more thought was put into successful onside kicks, they could get a success rate better than 20%.

The success rate on "unexpected" onside kicks is over 50%. The problem is that most of them happen during those down by 4 with 35 seconds left situations, where the defense is trying to defend one.

 

[patchick]

These models really won't work in real time until the computing power is available that can instantaneously take into account every variable from down and distance, field position and time of game to the tendencies of every player likely to be on the field from both teams in similar situations, the experience of both teams in similar situations, a database of all nfl teams in similar situations, the possible impact of not making the play, the weather, the stadium and a dozen other intangibles.

Yep, I don't even think that automated decision making like that is even part of the goal for the foreseeable future. Rather than creating a perfect computer model, the goal as a coach is to use the data to adjust your own mental model to counteract the irrationalities that come from conventional wisdom...and from being human. IMO, it's pretty clear that Belichick is very active in pursuing this. After watching the Pats so much, I find myself astonished watching other teams punt on 4th and 1 from the opponent's 45.

As a draftnik, I'm particularly interesting in how they number-crunch draft strategy, like the example of taking advantage of the 1-round bump for delayed picks. It seems to me that while the Pats are very aggressive with risk on the field, they take a much more risk-averse approach to the draft -- e.g. trading down from the 1st round so often.

(BTW, for those who get tired of hearing about draftee measurables, this is what that's about for me. It's just a way to collect data to try to find tendencies that predict selections and success. E.g., the fact that the 2 drafted WRs with the lowest shuttle times since 2002 are also the only 2 WRs who have made an impact with the Patriots as rookies is interesting, no?)

 

[patchick]

You have to read Carroll's book, The Hidden Game of Football (see my posts above).

The one you called "densely and poorly written"? OK, I'll try...

 

[Synovia]

IThe authors evaluate options solely on expected points on that possession, with ZERO weight given to time of possession and opponents' ensuing field position

Expected points compensates for opponent field position. IE, if you turn the ball over at the 20, you go from 5 expected points, to your opponent having 2 expected points.Loss of 7 expected points (numbers made up, but concept is correct)

 

[Synovia]

As a draftnik, I'm particularly interesting in how they number-crunch draft strategy, like the example of taking advantage of the 1-round bump for delayed picks. It seems to me that while the Pats are very aggressive with risk on the field, they take a much more risk-averse approach to the draft -- e.g. trading down from the 1st round so often.

I don't think the trading down actually is risk-adverse. I think they difference between a low 1st rounder and high 2nd, isn't the potential, but the chance they hit it. IE, they're amassing higher variance players who happen to cost much less. Its the same sort of thing the red sox did with smoltz and penny (which didn't work out): you've got a pretty full roster, so guys with high potential (even if they have low chance of hitting it) are much more valuable than guys who should be decent players but won't be good ones.


Plus, they're taking advantage of a serious market inefficiency when they're trading this year's 2nd for next year's 1st.

 

[SunnyDenmark]

another point about morale.

I am no 'westpoint' graduate - but morale often times is grossly overlooked in the history of battles (in favor of equipment, tactics, manpower advantage, terrain, etc). It is almost like morale is an afterthought. But it is very important. A demorilized enemy can be a very ineffective fighting force.

How does this relate to NFL? In regards to 4th down conversions - it was stated how about the affect on morale after the attempt? If 4th down is NOT converted by your teams offense: sure you are demorilized BUT the most directly involved (offense) gets to go off the field and 'regroup'. Your teams defense - whom are your 'reinforcements' -are fresh and motivated. However, if the 4th down conversion is sucessfull for your offense, the other teams defense is demorilized AND they also have to stay on the field to continue fighting for at least another 3 downs. So lets say even if it was a 50% / 50% proposition (at say the 50 yard line) - the morale 'penalty' is more severe on the other team. That in itself is an advantage even if the attempt is successfull only half the time.

 

[patchick]

Expected points compensates for opponent field position. IE, if you turn the ball over at the 20, you go from 5 expected points, to your opponent having 2 expected points.Loss of 7 expected points (numbers made up, but concept is correct)

Right -- except as I read it, in this study they only considered the opponent's expected points when the drive ended in a turnover. Field position after a punt, missed FG or failed 4th-down conversion wasn't a factor.

 

[PatsFanSince74]

Yep, I don't even think that automated decision making like that is even part of the goal for the foreseeable future. Rather than creating a perfect computer model, the goal as a coach is to use the data to adjust your own mental model to counteract the irrationalities that come from conventional wisdom...and from being human. IMO, it's pretty clear that Belichick is very active in pursuing this. After watching the Pats so much, I find myself astonished watching other teams punt on 4th and 1 from the opponent's 45.

As a draftnik, I'm particularly interesting in how they number-crunch draft strategy, like the example of taking advantage of the 1-round bump for delayed picks. It seems to me that while the Pats are very aggressive with risk on the field, they take a much more risk-averse approach to the draft -- e.g. trading down from the 1st round so often.

(BTW, for those who get tired of hearing about draftee measurables, this is what that's about for me. It's just a way to collect data to try to find tendencies that predict selections and success. E.g., the fact that the 2 drafted WRs with the lowest shuttle times since 2002 are also the only 2 WRs who have made an impact with the Patriots as rookies is interesting, no?)

Great comments.

For the NFL, this science is still in its infancy.

As you rightly observed in an earlier post, it's a very different matter in MLB where you have the limits of innings and a ginormous database on just about every aspect of the game and the tendencies of every player, including a strike zone divided into 16 blocks or something. There are enough data, for example, on what Batter A has done with a 2--2 count with runners on second and third with two outs in the bottom of the ninth on a calm, clear day with temperatures in the mid 70's at home against L and R handed pitchers and enough data on what Pitcher B has done against Batter A or in similar situations to make predictions more meaningful. Baseball has the benefit of a much more clearly defined set of data and situations that has been gathered now for decades. As large as the ultimate database is, it is well within the capacity of today's more powerful home or office computers, without having to turn to a mainframe. Don't get me wrong, this isn't "simple" to model; it's just that the variables have been clearly defined and tracked for a long time.

For the NFL, the data are much "softer" and, in the past, have been regarded as almost unanalyzable. 22 different players have a direct impact on the outcome of every play; is the Safety likely to blitz? How does our Back do on picking up that blitz? How has our line done against their nose tackle in the past? So far today? etc., etc. Down and distance, field position, game situation, current score, each team's risk profile for the game (is it a "must win?"), how each of those 22 individual players is likely to perform, field conditions, weather. All of these variables interact with each other to present a much more complex decision model.

The computing capacity required to turn what used to be regarded as "soft" and/or "unanalyzable" data into information in real time is probably beyond what most NFL teams have on board today. Our home PC's could probably clunk out an answer after twenty or thirty minutes; a mainframe could probably do it in a couple of minutes, but there are 40 seconds between plays, with at least 20 of them needed to get the players on the field and up to the line.

I have a nifty monte-carlo program that runs all sorts of fun simulations for me, but I have time to fix my lunch and bring it back to my desk in the time it needs to do 100,000 even simple calculations. What we're talking about here is a model with hundreds, if not thousands of variables that need to be run against each other many thousands of times in order to come out with an expected outcome and the shape of the distribution around that outcome...and then boil the whole thing down to a set of simple outputs that the HC can digest in five seconds and make a decision that accounts for the risks involved and his own "gut."

 

[makewayhomer]

almost every 4th down study says the same thing: coaches SHOULD be going for it a lot more than they should. coaches aren't aggressive enough

and there is a reason for this: coaches are so afraid to lose their jobs that they stick to conventional wisdom, b/c that won't get them fired. going for it on 4th and inches from your own 30, though, might well get a coach fired if it doesn't work.

or any other example from the Buffalo/Jets game the other week. tie game, Buffalo driving with less than a minute to go. they advance far enough for a 50ish yard FG attempt, and then what do they do? they stop trying to advance more and kill the clock!!!

they settled for a coin flip FG instead of actually trying to advance more. move the ball another 10-15 yards and your win % skyrockets. but they didn't want to take that "chance" I guess b/c they were afraid of 1 horrible outcome (TO). so, so , so dumb. but this way, when the kicker misses the FG, it's his fault, and not the coaches!!!

not getting fired has become more important than doing the smart thing