NFL draft success analysis

[Deus Irae]

Nothing really surprising here. When you eliminate 2000-2004, you cut

Seymour
Branch
Warren
Wilfork
Light
Graham
Wilson
Watson


from the equation. Then, since you're using games played in the analysis, you short change the latest draft classes, especially if a player in them gets hurt one year. Since the Patriots lull in drafting hit mostly from 2006-2009, this is pretty much exactly what people should expect to see in an analysis of this sort.

Window choice is huge in these types of discussions, which is why we've often had problems with 'cherry picking'.

 

[Mike the Brit]

RayClay ... has this study been discussed?

NFL daft: A partial history of draft day's terrible decisions and total failures - SBNation.com

It's absolutely amazing (and as reliable as any statistical study is likely to be).

 

[cstjohn17]

RayClay ... has this study been discussed?

NFL daft: A partial history of draft day's terrible decisions and total failures - SBNation.com

It's absolutely amazing (and as reliable as any statistical study is likely to be).

This part was pretty funny

"The rest of us mortals are confined by the oppression of common logic, our feet held to the ground by laws that stood for millions of years before anyone came around to write them. Bill Belichick is confined by none of them. He is a wizard whose command of elements real and abstract stretches beyond time, space, or any other dimension that the universe fecklessly tosses at his feet, like a single sandbag against a high tide. One day he will run out of idle curiosity, leave football, teleport to the Seahorse Nebula, and cook stews in the craters of unseen planets until animals crawl out. For now, he is content to outsmart your favorite team into oblivion."

 

[SalemPats]

I would like them to do a study on late picks and undrafted FAs or acquisitions. Look at our offense when we broke records.

 

[Mike the Brit]

Yeah!

He's a young guy and he's entitled to have his fun (this is the internet) but the stats are serious (much better than the National Football Post article, so far as I can see).

And ... wow! You don't have to be a rocket scientist to see an outlier. Below average drafting. Most victories in the NFL. What can you say?

 

[zimmy]

The metric is flawed. You can't just divide starts by the amount of picks made. The Pats strategy over the past few years has been to trade down and take more shots at lower percentage picks, but ultimately all team start off with 1 pick per round. The Pats shouldn't be penalised in that way.

If they started off with 7 picks, picked 7 times and got 4 starters it shouldn't be different from trading down, picking 11 times and getting 4 starters. The result is the same.

Plus as others have pointed out, its harder to crack the line up in a team that has started from a better base. For example, the Browns drafted Brandon Weeden and had to play him out of necessity and he was terrible. We drafted Shane Vereen and had him red shirt his first year while he learnt the system. This metric makes Weeden a far better pick but I know who I'd rather have.

Not to mention that we have one of the worst starting draft positions every year.

 

[Sicilian]

And ... wow! You don't have to be a rocket scientist to see an outlier. Below average drafting. Most victories in the NFL. What can you say?

That seems to be an indictment on the method used to assess draft grades to me. Either that or it tries to view the draft in a vacuum instead of what it is in reality: a piece of team building that can't be judged independent from other factors. Example: Drafting a rookie that gets beat out by a veteran addition for starts is viewed as a "bust". When in reality the team still has that position covered, so who cares where the player came from?

 

[Danger Zone]

Example: Drafting a rookie that gets beat out by a veteran addition for starts is viewed as a "bust". When in reality the team still has that position covered, so who cares where the player came from?

Mmm.. like drafting Mallet say?

 

[Mike the Brit]

That seems to be an indictment on the method used to assess draft grades to me. Either that or it tries to view the draft in a vacuum instead of what it is in reality: a piece of team building that can't be judged independent from other factors. Example: Drafting a rookie that gets beat out by a veteran addition for starts is viewed as a "bust". When in reality the team still has that position covered, so who cares where the player came from?

No.

This is by far the best piece of objective data-gathering I've ever seen on the Pats. What it says as clearly as it's possible is that the Patriots aren't very good at drafting but are fantastic at getting results.

While there is a surprisingly strong correlation between drafting and results for other teams -- enough to suggest that it's one of the most important determinants of overall success -- the Patriots are way, way better than they should be if they were within the bounds of normality.

Maybe BB's relative lack of draft success is the price you have to pay for doing things the BB way. Maybe they would have 5 rings if only they'd have drafted a bit better.

It's up to you to spin it how you like.

 

[Rob0729]

No.

This is by far the best piece of objective data-gathering I've ever seen on the Pats. What it says as clearly as it's possible is that the Patriots aren't very good at drafting but are fantastic at getting results.

While there is a surprisingly strong correlation between drafting and results for other teams -- enough to suggest that it's one of the most important determinants of overall success -- the Patriots are way, way better than they should be if they were within the bounds of normality.

Maybe BB's relative lack of draft success is the price you have to pay for doing things the BB way. Maybe they would have 5 rings if only they'd have drafted a bit better.

It's up to you to spin it how you like.

How is it objective? It is flawed. Based on some of these metrics, they consider Matt Schaub was a better draft pick than Andrew Luck or Russell Wilson (regardless where they were drafting) because he has more starts than either one of them. Not because he is a better pick, but he has been in the league since 2004 and has had the ability to start more games than two potentially elite QBs of the future who have only been in the game for two years. How is that an objective analysis?

This whole analysis is totally flawed because it gives more weight to players who have been in the league longer than players of the same quality with fewer years in the league. That means a player drafted in 2004 means more to the results of this study than a player drafted in 2013. There is no weighting to make a player who has been in the league count as much as a player who has been in the league for two.

For example, the Jets are much higher than they should be because they absolutely hit it out of the park in 2007 and 2008 and have absolutely sucked at drafting between 2009-2011. If years were reversed, they would be far lower on the list.

If this was written to go into an academic journal, it would be rejected because of faulty data and the information was misleading because there was no weighting to account for the fact that players who have been in the league for longer could have far more starts than a player drafted far better than him in subsequent years.

BTW, this isn't spin. As a person who spent many years dealing with statistics and marketing data, it is a simple fact that it is just faulty data and I explained why. Unless you think Matt Schaub was a far better player and draft pick than either Andrew Luck or Russell Wilson.

 

[Mike the Brit]

How is it objective? It is flawed. Based on some of these metrics, they consider Matt Schaub was a better draft pick than Andrew Luck or Russell Wilson (regardless where they were drafting) because he has more starts than either one of them. Not because he is a better pick, but he has been in the league since 2004 and has had the ability to start more games than two potentially elite QBs of the future who have only been in the game for two years. How is that an objective analysis?

This whole analysis is totally flawed because it gives more weight to players who have been in the league longer than players of the same quality with fewer years in the league. That means a player drafted in 2004 means more to the results of this study than a player drafted in 2013. There is no weighting to make a player who has been in the league count as much as a player who has been in the league for two.

For example, the Jets are much higher than they should be because they absolutely hit it out of the park in 2007 and 2008 and have absolutely sucked at drafting between 2009-2011. If years were reversed, they would be far lower on the list.

If this was written to go into an academic journal, it would be rejected because of faulty data and the information was misleading because there was no weighting to account for the fact that players who have been in the league for longer could have far more starts than a player drafted far better than him in subsequent years.

BTW, this isn't spin. As a person who spent many years dealing with statistics and marketing data, it is a simple fact that it is just faulty data and I explained why. Unless you think Matt Schaub was a far better player and draft pick than either Andrew Luck or Russell Wilson.

No.

"For most of my analysis, I've relied upon a couple of pre-existing statistical models. The first was Pro-Football-Reference's Career Approximate Value (CarAV) score. For starters, Approximate Value (AV) is Pro-Football-Reference's attempt at reducing a player's overall value to a single number. This season, Peyton Manning and Richard Sherman led the league in AV with 19. LeSean McCoy had 15, Tony Romo had 13, et cetera. This stat should be handled carefully -- "approximate" is right there in the dang name -- but it's quite useful for comparing the value of large groups of players across different positions.

CarAV represents the AV of a player throughout every season of his career, but weights it so that his best seasons count a little more. That's swell for our purposes, since a player's maximum potential is what we're really after."

CarAV is not perfect, but it doesn't make the silly mistakes you say.

 

[Rob0729]

BTW, I am admitting the Pats' 2006-2009 drafts were bad for varying reasons. Some was bad draft picks (2006 for example) and some mostly late round picks (2007).

But I would put the Pats' 2010-2013 drafts up against most team in the league. Unfortunately, this survey consider those drafts of lesser importance than 2004-2007 based on the data collection. If the Pats had the same quality drafts in 2004-2007 as they did in 2010-2013 and had the drafts from 2006-2009 between 2010-2013, the Pats would be considered a much better drafting team.

Again, not spin. Simple facts of this survey. A player who has been in the league 10 years has the opportunity to start more games than player who has been in the league for two years.

 

[Rob0729]

No.

"For most of my analysis, I've relied upon a couple of pre-existing statistical models. The first was Pro-Football-Reference's Career Approximate Value (CarAV) score. For starters, Approximate Value (AV) is Pro-Football-Reference's attempt at reducing a player's overall value to a single number. This season, Peyton Manning and Richard Sherman led the league in AV with 19. LeSean McCoy had 15, Tony Romo had 13, et cetera. This stat should be handled carefully -- "approximate" is right there in the dang name -- but it's quite useful for comparing the value of large groups of players across different positions.

CarAV represents the AV of a player throughout every season of his career, but weights it so that his best seasons count a little more. That's swell for our purposes, since a player's maximum potential is what we're really after."

CarAV is not perfect, but it doesn't make the silly mistakes you say.

Maybe we are looking at two different surveys. The National Football Post referenced in the OP does not use that formula. That one uses straight accumulation of total starts by all the draft picks from 2004 through 2013. That is an immensely flawed survey. That is what I am commenting on.

I see you put your own link, but that one is flawed too because it only looks at 2004-2008 which includes three of the worst drafts of the Belichick era and none of their best drafts of the Belichick era (2003, 2010, etc.). I don't know how looking at a sample size of Belichick's drafts that include all of Belichick's worst drafts (except for 2009) and none of his best ones tells you anything either.

EDIT: I see now I was responding to your post that seemed to responding to your link you posted, but as I said the sample size in your article is subjective because it basically only included the lowest point of the Patriots drafts under Belichick and none of the high points.

 

[unoriginal]

I think this is key.

They Patriots explicitly use a strategy of maximizing expected value of draft picks. They would rather have three picks each with a 30% chance of success than one with a 75% chance of success.

Just to bust your balls for a moment.

Three picks each with a 30% chance of success means each has a 70% chance of failure.

The chance that all three fail is therefore 70% ^ 3. That winds up being a 34.3% chance that all three of those draft picks crap out.

Versus 25% chance of failure for the one draft pick.

So that's not a great example of the Patriots' draft wisdom.

 

[PlainOldEd]

I've seen this argument a number of times, and have trouble supporting it.

The Patriots commonly lead the league in undrafted rookies making the team.

Last year's team included James Develin, Kenbrell Thompkins, Zach Sudfeld, Will Svitek, Chris Barker, Josh Kline, Joe Vellano, AJ Davis, and Ryan Allen. Not positive about all those guys, but most were originally undrafted. Hard to argue that starting players drafted by another team wouldn't have made the Patriots 2013 roster.

Making the team, but not necessarily starting.

 

[RayClay]

RayClay ... has this study been discussed?

NFL daft: A partial history of draft day's terrible decisions and total failures - SBNation.com

It's absolutely amazing (and as reliable as any statistical study is likely to be).

I'll bet Belichick has used statistics more than anyone, but to me the problem is not that statistical models are wrong, it's that there is no value system that can decide what is "better" and assign a statistical value to that.

There's no doubt that playing as a team, with good chemistry, working the salary cap differently every year depending on the demographic makeup (age contracts coming due etc.) is probably more important than acquiring raw talent, otherwise the worst teams would flip to being the best teams every few years, yet they don't.

You simply don't have the same shot at talent picking #29 or so every year.

I'm not a big believer in broad statistics applying to the game, or the draft in football. You can trade for a 300 hitter or pitcher with a low ERA and usually know what you're getting (except, when they pitched in a huge park and moved to Fenway.

In Football you can have a bottom 10 defense and offense and win the Super bowl, because the last yard is worth more then the preceding 99 on the field.

I imagine the Pats spend more time coming up with tiny relationships they can measure to find any that might yield actual useful information than trying to make huge statistical models full of variables, all of which need to have values assigned to them, decreasing their actual reliability as useful measure sof one of many factors in building a team.

 

[DarrylS]

Many times on the Patriots there are no "job openings" so a rookie often does not have to be a "starter" right away, consider Jamie Collins according to this chart he would probably be credited with two starts..

Jamie did not start because he was not good, but because he was developing... so a lot of these "starts" need qualification.. overall this chart and method is too simple to explain all of the subtleties...

 

[TruthSeeker]

Several things:

- If you did this survey over the last five years, the Pats would probably be in the top third of the league. It was 2006-2009 where the Pats had the least draft success. From 2010-2013 have been solid draft.

- The fact that the Pats were successful in the draft in the last four years, but unsuccessful between 2006-2009 penalizes them in the number of starts category. A stud player drafted in 2006 is going to have more starts than a stud player in 2013. It is a simple fact. Say the Pats drafted Chandler Jones in 2006, he would have 128 starts (assuming he was healthy) in this study rather than 30 starts he has now even though he would be the same player. That category is definitely skewed to favor draft picks made ten years ago over last year.

- Most games started by draft picks is a misleading stat. When you have a Super Bowl contending team, it is difficult for a draft pick to crack the starting line up. There are many bottom tier teams starting players they drafted that wouldn't even make the Pats' 53 man roster. You see many of the teams most successful in this category were below .500 for most or all of the time period (2004-2013) surveyed. That tells you that this is a bad way to judge success in drafting.

- The Pats are penalized by some of these rankings for trading down. They use percentage of draft picks that go to the Pro Bowl. This is where they penalize the Pats for trading down. So if the Pats had two first rounders and one turned into a Pro Bowler, they would have a 50% success rate. If the Pats had two first rounders and traded one down for three picks and only the first rounder makes the Pro Bowl, the success rate turns into 25% even if the three players who were traded for were good players and a better value than one first rounder. This survey benefits a team like the Jets who tends to put their eggs in one basket and trade large portion of their draft for one player than a team like the Pats who values quantity of picks.

- The table that judges the success rate in the first three rounds is very skewed against the Pats since they trade out of the first round a lot.

I think the Pats have been above average drafting over the decade with a bad period early in that decade. This study penalizes them for this while forgives a team like the Steelers who drafted well in 2004-2008, but god awful in the last three years.

Agreed. The methodology, while not awful, is also not good. It's a superficial analysis that attempts (but fails) to provide depth. Other draft analysis posted on this board in the past couple of years has the Patriots the best drafting team in the NFL over the past decade. Clearly using different criteria for success gets totally different results.

Of course, draft analysis is hard, not easy. At least this writer put in some effort to do some statistical analysis even though, in the end, it proved practically nothing. I'd make out the correlation of the analysis to the actual draft value is perhaps .20 or .30 (on the -1 to 1 scale).

 

[Fencer]

1. The Patriots probably started out with much less draft "currency" than any other team, because:

A. They had the worst average draft position of any team due to their great success.
B. They lost one 1st round pick to ... well, you know.
C. They traded two 2nd round picks for vets.

2. They recovered some draft currency by trading into future seasons.

3. They traded down a lot and up sometimes.



On the whole, I'd guess that the Patriots' average pick, under any weighting system you'd care to use, was greatly below the league averages, and non-trivially below any other team's.

 

[Mike the Brit]

1. The Patriots probably started out with much less draft "currency" than any other team, because:

A. They had the worst average draft position of any team due to their great success.
B. They lost one 1st round pick to ... well, you know.
C. They traded two 2nd round picks for vets.

2. They recovered some draft currency by trading into future seasons.

3. They traded down a lot and up sometimes.



On the whole, I'd guess that the Patriots' average pick, under any weighting system you'd care to use, was greatly below the league averages, and non-trivially below any other team's.

This is factored in to the SBnation article by looking at the picks actually made by teams, valuing them according to a draft value chart, and then dividing the value of the total return by that number. Believe me, the guy who did this isn't stupid.