Success at drafting ( a metrics inventory)

[fester]

One of the constant pissing matches is what defines successful drafting. There are some posters in this forum (and I assume any other professional sports fans forums) that maintain anything less than hitting on 100% of picks is absymal failure that can only be corrected if the Patriots hire them based on their astute internet fan board scouting. I think that strawman is unreasonable. But there is a wide array of opinions as to what constitutes "successful" drafting.

Let's list as many different success metrics as possible with potential upsides and downsides of using that particular definition.

1) 1st year rookie starters (credit DW Toys) --- % of drafted rookies who start 5 games or more OR 10 games or more during their rookie year. The more restrictive criteria produces an 18% hit rate in 2011. The upside of this metric is that it measures immediate impact. The downside is that it is very imprecise as it does not account for team quality (bad teams with horrendous depth will have more rookie starters, all else equal, than good deep teams (see the 2007 Patriots for instance). Other downside is that it does not account for slow to develop positions (Aaron Rodgers would be graded as "non-successful" using this metric)

2) 2nd contract evaluation --(my preferred method) --- % of drafted rookies who receive multi-year contract extensions that extend a player past their RFA time and into their UFA eligbility time) where the contract extension average annual value is equal to or greater than either a 1st round RFA tender, or a looser metric a 2nd round RFA tender. Upside is that this is a team's attempt to value a player entering their prime against market values. The downside is that this is a slow process, at least three years from the draft in question.

3) Starters per year (Good draft hit rates: Lower than you think - AFC South Blog - ESPN) Interesting ESPN article on two GMs (Titans and Colts) who think they need to find 3 solid starters per year on average. This neglects a bit of draft pick efficiency but looks strictly at effectiveness. In a single year, it has the downsides of #1 in neglecting team quality (bad teams should have more marginal hits because the rest of the roster sucks) but it compensates if there is a multi-year averaging.

4) Expected Performance outcomes (good example here NFL Draft Success by Round) A variety of models that look at a players' career achievements versus the expected performance for that draft position. This rewards teams who have low draft ammunition but good results while downgrading teams that are picking at the tops of the rounds (easier to find good players picking first instead of last).

What other metrics of success are there, and what are their upsides and downsides.

 

[everlong]

I think breaking it down by round and even half round in the case of the first round and comparing it to how other teams do is a fair metric but it takes the first contract to really assess the value. Some players flash year one and fade. Some take a year or two and then shine.

I do not think you can lump all picks together and say well team A hit on 50% of their draft picks from the period of 3-6 years ago and team B only hit on 33%. Team A drafted 4 players in the top 15 during that time period and team be never drafted higher than 24.

It also doesn't make sense to lump rounds together. A 5th round pick who spends 4 years on your roster as a core special teams player and plays a sub package role on offense or defense is a complete success. A first rounder with the same resume is viewed as a bust.

I think assigning a scoring system by round based on performance criteria is the way to go.

I'd also subtract a lot of points the higher the bust. Kevin O'Connell would merit a big loss vs an Oscar Lua just a no points awarded. Injuries would be the hard part of this. You really can't fault a team for drafting a Robert Edwards and then the poor kid gets hurt in an event that never should have been and he showed such promise (bitterness).

 

[jmt57]

One of the constant pissing matches is what defines successful drafting. There are some posters in this forum (and I assume any other professional sports fans forums) that maintain anything less than hitting on 100% of picks is absymal failure that can only be corrected if the Patriots hire them based on their astute internet fan board scouting. I think that strawman is unreasonable. But there is a wide array of opinions as to what constitutes "successful" drafting.

Let's list as many different success metrics as possible with potential upsides and downsides of using that particular definition.

1) 1st year rookie starters (credit DW Toys) --- % of drafted rookies who start 5 games or more OR 10 games or more during their rookie year. The more restrictive criteria produces an 18% hit rate in 2011. The upside of this metric is that it measures immediate impact. The downside is that it is very imprecise as it does not account for team quality (bad teams with horrendous depth will have more rookie starters, all else equal, than good deep teams (see the 2007 Patriots for instance). Other downside is that it does not account for slow to develop positions (Aaron Rodgers would be graded as "non-successful" using this metric)

2) 2nd contract evaluation --(my preferred method) --- % of drafted rookies who receive multi-year contract extensions that extend a player past their RFA time and into their UFA eligbility time) where the contract extension average annual value is equal to or greater than either a 1st round RFA tender, or a looser metric a 2nd round RFA tender. Upside is that this is a team's attempt to value a player entering their prime against market values. The downside is that this is a slow process, at least three years from the draft in question.

3) Starters per year (Good draft hit rates: Lower than you think - AFC South Blog - ESPN) Interesting ESPN article on two GMs (Titans and Colts) who think they need to find 3 solid starters per year on average. This neglects a bit of draft pick efficiency but looks strictly at effectiveness. In a single year, it has the downsides of #1 in neglecting team quality (bad teams should have more marginal hits because the rest of the roster sucks) but it compensates if there is a multi-year averaging.

4) Expected Performance outcomes (good example here NFL Draft Success by Round) A variety of models that look at a players' career achievements versus the expected performance for that draft position. This rewards teams who have low draft ammunition but good results while downgrading teams that are picking at the tops of the rounds (easier to find good players picking first instead of last).

What other metrics of success are there, and what are their upsides and downsides.

One slight change I would personally make to the metrics above is to use total numbers rather than percentages. The reason for this is that every team starts out with the same number of draft picks every year, seven. If one team is fond of trading up, then their percentage should be expected to be higher; if another team prefers to trade down then their percentage should be expected to be lower. You could end up where the former team has fewer 'success' picks, yet grades out better than the latter team who has more 'success' picks but a lower percentage.

This of course is going to be thrown off by trades; which is another debate in itself. Do you consider the player that was traded for in draft analysis? After all, it's still player evaluation: a college player in comparison to a veteran, just like when you draft a player. Should trading a draft pick for Greg Lewis, Alex Smith or Albert Haynesworth be considered a draft bust? Should utilizing draft picks for Wes Welker or Randy Moss be considered a draft 'success'?

Trades in particular are going to skew single year grades, but if you look at a larger sample size (decade by franchise, HC or GM history) then you should get a much better picture regardless of how you handle trades.

The only bad thing about utilizing rounds is that pick number 1 is treated equal to pick number 32, while picks 32 and 33 are treated differently. It's better than treating all draft picks the same, but it could still be tweaked a little bit (e.g., with a formula using the draft trade value chart) - though that would start to get rather complicated.


Overall I really like your idea; perhaps some combination of those metrics might be best. It has always seemed rather crazy to me when so many people complain about draft 'busts' when nobody (other than you and DW) seems to have any idea of what number is good, average or bad, or how to define/grade a draft pick.

 

[ay-yo]

One of the constant pissing matches is what defines successful drafting. There are some posters in this forum (and I assume any other professional sports fans forums) that maintain anything less than hitting on 100% of picks is absymal failure that can only be corrected if the Patriots hire them based on their astute internet fan board scouting. I think that strawman is unreasonable. But there is a wide array of opinions as to what constitutes "successful" drafting.

Let's list as many different success metrics as possible with potential upsides and downsides of using that particular definition.

1) 1st year rookie starters (credit DW Toys) --- % of drafted rookies who start 5 games or more OR 10 games or more during their rookie year. The more restrictive criteria produces an 18% hit rate in 2011. The upside of this metric is that it measures immediate impact. The downside is that it is very imprecise as it does not account for team quality (bad teams with horrendous depth will have more rookie starters, all else equal, than good deep teams (see the 2007 Patriots for instance). Other downside is that it does not account for slow to develop positions (Aaron Rodgers would be graded as "non-successful" using this metric)

2) 2nd contract evaluation --(my preferred method) --- % of drafted rookies who receive multi-year contract extensions that extend a player past their RFA time and into their UFA eligbility time) where the contract extension average annual value is equal to or greater than either a 1st round RFA tender, or a looser metric a 2nd round RFA tender. Upside is that this is a team's attempt to value a player entering their prime against market values. The downside is that this is a slow process, at least three years from the draft in question.

3) Starters per year (Good draft hit rates: Lower than you think - AFC South Blog - ESPN) Interesting ESPN article on two GMs (Titans and Colts) who think they need to find 3 solid starters per year on average. This neglects a bit of draft pick efficiency but looks strictly at effectiveness. In a single year, it has the downsides of #1 in neglecting team quality (bad teams should have more marginal hits because the rest of the roster sucks) but it compensates if there is a multi-year averaging.

4) Expected Performance outcomes (good example here NFL Draft Success by Round) A variety of models that look at a players' career achievements versus the expected performance for that draft position. This rewards teams who have low draft ammunition but good results while downgrading teams that are picking at the tops of the rounds (easier to find good players picking first instead of last).

What other metrics of success are there, and what are their upsides and downsides.

I know these aren't your metrics, but IMO:

1) I think starts is too restrictive. You want production of some sort, but doesn't have to be starts.

2) Again no need to restrict it to getting a specific tender. If a player gets a 2nd contract it proves your drafting at least competent NFL players.

3) 3 starters seems a bit high, I'm not sure how many teams hit this mark

I agree with JMT. Some combination of methods 1,2 and 3 would make the most sense.

JMT: I always consider players traded for in draft analysis.

 

[fester]

One slight change I would personally make to the metrics above is to use total numbers rather than percentages. The reason for this is that every team starts out with the same number of draft picks every year, seven.....

This of course is going to be thrown off by trades; which is another debate in itself. Do you consider the player that was traded for in draft analysis....

Trades in particular are going to skew single year grades, but if you look at a larger sample size (decade by franchise, HC or GM history) then you should get a much better picture regardless of how you handle trades.

The only bad thing about utilizing rounds is that pick number 1 is treated equal to pick number 32, while picks 32 and 33 are treated differently. It's better than treating all draft picks the same, but it could still be tweaked a little bit (e.g., with a formula using the draft trade value chart) - though that would start to get rather complicated.


Overall I really like your idea; perhaps some combination of those metrics might be best. It has always seemed rather crazy to me when so many people complain about draft 'busts' when nobody (other than you and DW) seems to have any idea of what number is good, average or bad, or how to define/grade a draft pick.

JMT --- thanks for the compliment, but this is much more a metrics than a methods post. Let's get to the meat of your comments:

a) Not all teams have 7 picks, some will be allocated 7 (like the Patriots this year) and others could be allocated up to 11 (Cleveland Browns) via the comp pick system. But that is a minor quibble.


b) Depends on what you are attempting to analyze -- is it an attempt to evaluate the effectiveness of a scouting system to find college players than disregard trades and only look at college players selected. If the goal is to identify team building effectiveness, than include the trades as part of the comprehensive whole. But that is a methods paper not a metrics inventory that I had enough time to write about because my daughter was home sick today with an ear infection

c) I completely agree, there is minimal difference in reality between #32 and #33 or #64 and #65 while there is usually a talent cliff somewhere between #33 and #64. Using a draft point value system of some sort smooths out a step function into a something looks vaguely like a continuous function. I think it allows analysts to look at drafting efficiency because all else being equal, a team like Indianaoplis with their 4,041 allocated draft points should have a better draft than the Patriots with their allocated 1,086 draft points or 2,121 currently held draft points.

d) My main interest is identifying how people define "successful" drafting. You raise a good alternative method that focus mainly on the end result --- how many good players did the management team find through the college player process.

How else do people define successful drafting?

• discrepencies between external ranking list and actual draft position for "Value"
• Team needs versus positions drafted
• Binky orgasm

 

[fester]

3) 3 starters seems a bit high, I'm not sure how many teams hit this mark

I agree with JMT. Some combination of methods 1,2 and 3 would make the most sense.

I think most teams have to hit that mark as there are between 22 and 28 "starters" or "starter quality" players on the game day 46. (Base 11 on offense and defense, 3rd WR, 3rd down RB, Nickel Back, Kicker, Punter, Special Teams Ace 1 and 2?)

Averaging 3 quality starters per year would mean a team turns over their starting line, on average every eight or nine years or so. That actually seems slightly long for turnover, as some positions may turn over twice before another position turns over once as the Pats have had some keystone players and their contract structures identify current cornerstone players in Brady, Mankins, Wilfork, Mayo, and until his retirement rumors, Light.

And if a team does not consistently hit 3 or more quality starters per year over a several year span like the Pats missed from 2006 to 2008 in the draft, there is a gap in the team's age, talent and salary structure of young, very talented 2nd contract players that is made up by a combination of putting more young players into high snap count roles that they may or may not be ready for, and acquiring more veterans from other teams via free agency or trades. But in that case, the Pats are "stealing" other teams initial successes in identifying competent NFL players from the college ranks, so it evens out across the league.

 

[jmt57]

JMT --- thanks for the compliment, but this is much more a metrics than a methods post. Let's get to the meat of your comments:

a) Not all teams have 7 picks, some will be allocated 7 (like the Patriots this year) and others could be allocated up to 11 (Cleveland Browns) via the comp pick system. But that is a minor quibble.


b) Depends on what you are attempting to analyze -- is it an attempt to evaluate the effectiveness of a scouting system to find college players than disregard trades and only look at college players selected. If the goal is to identify team building effectiveness, than include the trades as part of the comprehensive whole. But that is a methods paper not a metrics inventory that I had enough time to write about because my daughter was home sick today with an ear infection

c) I completely agree, there is minimal difference in reality between #32 and #33 or #64 and #65 while there is usually a talent cliff somewhere between #33 and #64. Using a draft point value system of some sort smooths out a step function into a something looks vaguely like a continuous function. I think it allows analysts to look at drafting efficiency because all else being equal, a team like Indianaoplis with their 4,041 allocated draft points should have a better draft than the Patriots with their allocated 1,086 draft points or 2,121 currently held draft points.

d) My main interest is identifying how people define "successful" drafting. You raise a good alternative method that focus mainly on the end result --- how many good players did the management team find through the college player process.

How else do people define successful drafting?

• discrepencies between external ranking list and actual draft position for "Value"
• Team needs versus positions drafted
• Binky orgasm

I suppose I would place players into tiers for draft evaluation, but before we get to that I would say the tier/grade/number is going to be different for each year that a player is evaluated; that too should be taken into consideration.

1. Among the best in the league at his position.
   
2. Above average at their position.
   
3. Marginal starters & top backups/role players.
   
4. The rest that are on rosters.
   
5. Not on the roster.

The first group you could for example use being voted to the pro bowl, though that often turns into popularity and reputation; some other way of determining this top group may be better.

Let's use corners for an example. The first group may be the top eight or ten players in the league; the second is the next 30 or so; the third is the next 40 or so; the fourth group is about another 70 players.

But how do you determine who fits where? Number of snaps could be used, but as mentioned previously players on bad teams get graded too high and subs on good teams too low. PFF is about the only site that I know of that grades and ranks every player, and the flaws that can happen in their analysis is well documented.

You also really need to re-do this for each season a player has; otherwise you end up giving a player that makes one pro bowl the same grade as a future hall of famer.

One other question is how do you treat injuries? Do the Pats get a zero when assessing their choice of Sebastian Vollmer due to how little he played last year? If not, then how do you grade Shawn Crable as of the end of the 2009 season?

In addition, how do you grade a pick who goes on to do well elsewhere? Does a team deserve any credit at all for drafting a player that gets cut during his rookie training camp and goes on to do well elsewhere? Personally I would say no but others will disagree. And what about undrafted free agents? Shouldn't a team get some credit for signing an UDFA that never played for another team? Obviously they scouted and evaluated him just like they did with his teammates that were drafted.


I guess I would assign a number based on the tiers above (1-5, or 0-4), unique for each year of a player's career, and divide it by the number of years he played to get an average. Conversely you could just add each of those numbers for every year he was with his original team; this way the player with a long career with his original team scores a much higher draft grade than one that walks, has a short career, etc.; bonus draft grade points for a quality player with a lengthy career with his original team.




On the other hand if you just want a much simpler metric that would be easier for people to follow you could add up the number of starts per draft class per year, and rank all 32 teams that way. Similarly rather than number of starts you could look at what players got the most snaps on each side of the ball over the course of the year and label them as a team's starter that year; add up the number of starters each team has and rank the draft classes. If you want you can add in punters, kickers and long snappers and count them as starters for this exercise.