Sackseer: A Seer or Sack It? (An Early Evaluation)

by Requiem on Sep 4, 2011 4:21 AM CDT

So, the idea for this fanpost came about because of some discussion about Sackseer and how effective it really is at predicting sack performance. We seem to have a contingent here at BTB who seems to think that Sackseer has a probative value in predicting pass-rushing performance.

I wonder, how valid this belief is, though, given:

The supporter of the stat is a paid site (footballoutsiders.com) which could benefit financially if they're able to get the stat legitimized.
There's no peer review of the work nor ANY WAY TO PEER REVIEW the work, as it's a "proprietary formula"
The main variable which is related to success according to Sackseer is Missed Eligible Games in College. Not Performance, not speed, not strength. But number of games missed.
The person who came up with the formula is an attorney.

Now, this isn't necessarily a problem in coming up with knowledge, per se. But there are some other squirrelly things in the methodology that I've read which leads me to the conclusion that the author being a lawyer is a direct hindrance of trusting that the stat is actually effective. Among those are, the following:

The author is not statistically-saavy. The author states that:
The trends that SackSEER identifies for edge rushers drafted in the first two rounds also persist when later-round edge rushers are modeled. For instance, SackSEER would have identified Robert Mathis and Adalius Thomas as top edge rusher prospects. However, the trends are not quite as strong and the projections not quite as accurate (the seven-round regression yields R-Squared of .27). Nonetheless, "seven-round SackSEER" does have some interesting implications for the two-round model.

First, the seven-round model demonstrates that it is unlikely that SackSEER is a product of "data-mining bias"
Unfortunately, it doesn't do anything of the sort since the seven-round model also contains the first two rounds which has the data that the author used to create his formula. This fundamental lack of understanding of statistics leads me to suspect that the author not coming from a statistical (or heck even mathematical/engineering) background seriously hinders his methodology.

To be a good lawyer, you must be an advocate of a position.

To be a good scientist/statistician, you must NOT be an advocate of a position. You must let the data tell you what is correct and what is not.

Take a look at some of the lines in the quote above:
The trends that SackSEER identifies for edge rushers drafted in the first two rounds also persist when later-round edge rushers are modeled. For instance,
Then, he goes on to list some examples and state that the relationships are not as strong despite the fact that the R-squared drops from .42 which is a decent relationship to .27 which isn't that strong (and that's not taking into account that the real relationship outside the first two rounds is almost definitely worse that a .27 R-squared relationship).
It seems strongly to me that he's advocating for something here, rather than let the data tell him what's going on.

Anyways, those are just some of my concerns with the Sackseer stat and its methodology. But I thought it'd be interesting to actually look at the data and the predictions and see how well the predictions are doing in relationship to the actual data.

Now, I acknowledge that there's a danger in prematurely making conclusions about the data since it is SUPPOSED to predict 5 year sack numbers. But we can get an early sense of whether there's anything interesting in the predictions or whether there might be some red flags about the stat.

A caveat I want to offer when looking at the data is that I'm using pro-ration as a comparison with the prediction. Now, pro-ration is probably not the best method of predicting what the player will do in the future. After all, some of them will grow and become better players (and some haven't even played in games). However, there IS also the downside of a potential career ending injury and there's also the issue that these are 1st and 2nd round talents, so for the majority of them they SHOULD be having an impact their first year.

So, I'm going to go with pro-ration for now mainly since it's easily understandable and it's easy to implement. If you have a better idea, please feel free to suggest.

Here are the numbers:

Player	Prediction	2010 Sacks	Proration	Vertical	Short Shuttle	SRAM	Missed Games
Jerry Hughes	27.7	0	0	34.5"	4.15	0.55	3
Derrick Morgan	23.3	1.5	7.5	34"	4.43	0.59	1
Everson Griffen	22.8	0	0	34"	4.36	0.52	3
Brandon Graham	22.1	3.0	15.0	31.5"	4.25	0.62	4
Sergio Kindle	18.8	0	0	36.5"	4.53	0.48	7
Carlos Dunlap	16.1	9.5	47.5	31.5"	4.61	0.62	2
Jason Pierre-Paul	3.8	4.5	22.5	30.5"	4.67	0.44	26

Hmmm, it doesn't look good for sackseer at this time. The lowest two predictions are currently doing the best out of the pass-rushers predicted for.

Just for kicks, I thought I'd take a look at the other player drafted in the 2nd round and compare their raw numbers (remember, I don't have access to the real sackseer numbers)

Player	2010 Sacks	Proration	Vertical	Short Shuttle	Production	Missed Games
Koa Misi	4.5	22.5	38"	4.27	10.5 S in 38 G	14
Jermaine Cunningham	1.0	5.0	35"	?	18.5 S in 45 G	10
Jason Worilds	1.0	5.0	38"	4.29	15 S in 39 G	13

Misi looks somewhat comparable to Kindle. His Vertical is slightly higher, his shuttle time is much faster, but his production isn't as good, and his games missed is significantly worse. So guess about 18.8 prediction?

Cunningham also looks somewhat comparable to Kindle. But we can't really know since we don't have a Shuttle time on him.

Worilds looks like he should be the best of the bunch since his vertical is tied for the best, and his short shuttle is comparable to Misi, and his production is better, and his missed eligible games is about the same as Misi.

Another user-created commentary provided by a BTB reader.

Tweet Comment 43 comments | 1 recs |

Do you like this story?

Comments

The supporter of the stat is a paid site (footballoutsiders.com) which could benefit financially if they’re able to get the stat legitimized.

what? it’s not even a stat. Football Outsiders is about 1000% better than the typical football analyst in terms of trying to get things right and looking critically looking at their own work. Here’s what forester wrote this year.

Although it is too early to enter any definitive judgment on last year’s draft class, there is no question that this was a rough year for SackSEER. SackSEER’s most highly projected edge rusher, Jerry Hughes, played little in 2010; and SackSEER missed on whatever transformed Carlos Dunlap from a healthy scratch at midseason into the most productive rookie edge rusher by year’s end.

Most notably, however, SackSEER’s controversial 4.5 sack projection for Jason Pierre-Paul now looks silly. Pierre-Paul met his five-year projection in the space of just his rookie year as part of a heavy rotation with the prolific Justin Tuck and Osi Umenyiora. With 4.5 sacks as a rookie, Pierre-Paul is on pace to be slightly more productive than the average highly drafted edge rusher.

and here’s Football Outsiders self-assessment of the Lewin Career Forecast

Five years ago, Football Outsiders unveiled our first college quarterback projection system. It came to be known as the Lewin Career Forecast, since it was created by a college kid named David Lewin who now works for the Cleveland Cavaliers. The elements were simple: The LCF did a surprisingly good job of projecting the success of first- and second-round quarterbacks using just college games started and college completion percentage. It was so popular that references to the Lewin Career Forecast started showing up all over the media, sometimes even “referencing” entire paragraphs of my writing.

There’s only one problem: In the last couple years, the LCF hasn’t done so well. The formula predicted success for a number of flops including Kellen Clemens, Brady Quinn, Brian Brohm, and Matt Leinart. I detailed these issues in an ESPN Insider piece last week, but let me summarize here for those of you who don’t get ESPN Insider. From 1997 through 2005, there were 11 quarterbacks who:

were chosen in the first two rounds
had at least 33 games started in college
completed at least 58 percent of passes in college.

Out of these 11 quarterbacks, the worst was Byron Leftwich, who was good enough to lead a 12-4 team to the playoffs in 2005. However, the same baselines between 2006 and 2009 produce this list of quarterbacks: Matt Leinart, Brady Quinn, Kevin Kolb, John Beck, Brian Brohm, Chad Henne, Josh Freeman, and Pat White. OK, maybe we don’t consider White as a player who was drafted as a “conventional quarterback,” but still, that list has four flops, one success (Freeman), and two guys who we’re not sure about yet (Kolb and Henne). It’s a huge change from 1998-2005.

…the real issue is number of games started. Before 2005, games started were a strong clue as to whether scouts got it right or wrong on the top prospects. Since 2005, many quarterbacks with plenty of experience washed out while similarly accurate, but much less experienced quarterbacks like Aaron Rodgers and Joe Flacco have become successful NFL starters.

The phrase “before 2005” gets to the second criticism, which is that LCF is more descriptive than it is predictive.

We’re going to see whether Sackseer has any merit by it’s performance over time.

There’s no peer review of the work nor ANY WAY TO PEER REVIEW the work, as it’s a “proprietary formula”

I believe you are mischaracterizing how sackseer came about. My understanding is Football Outsiders didn’t develop it themselves. Nate Forester developed it and sent it to Football Outsiders. Football Outsiders effectively is the peer review.

Editor’s Note: If you are interested in publishing a guest column at Football Outsiders, please e-mail your idea or rough draft to info-at-footballoutsiders.com. Come up with something really good, like SackSEER or the Lewin Career Forecast, and we may even ask to publish it in our book.

again, I think skepticism is merited. But I don’t think you’re showing skepticism which is a position of ‘I don’t know’. You seem to have the attitude the you do know for certain … you know it’s wrong.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 8:21 AM CDT reply actions

Responses

what? it’s not even a stat. Football Outsiders is about 1000% better than the typical football analyst in terms of trying to get things right and looking critically looking at their own work. Here’s what forester wrote this year.

Fine, it’s not a stat. It’s a prediction. Thanks for being nitpicky. Also, just because something is better than garbage, does not mean it isn’t garbage itself. I like the FO guys fine (well, some of them), but there’s definitely an attitude around them which doesn’t match up with their “expertise.” I’ve critiqued some of their work before, and some of them have been extremely defensive about it (usually the guys who don’t know as much). Others like Tanier, Farrar, and others, seem to handle critique much better.

I believe you are mischaracterizing how sackseer came about. My understanding is Football Outsiders didn’t develop it themselves. Nate Forester developed it and sent it to Football Outsiders. Football Outsiders effectively is the peer review.

No, I’m not. I know that Nate Forester (the lawyer) developed it. I never said it was FO’s proprietary formula, did I?

And frankly, I haven’t been overly impressed with the statistical chops of a lot of the guys at FO, so you’ll excuse me if I don’t take them as a great source of peer review.

again, I think skepticism is merited. But I don’t think you’re showing skepticism which is a position of ‘I don’t know’. You seem to have the attitude the you do know for certain … you know it’s wrong.

You seem to have a problem yourself with skepticism. You keep harping on Sackseer-based numbers like it’s the holy grail of pass-rusher numbers. Some could say that you seem to lack ANY skepticism what with the amount of times you throw around Sackseer-based numbers.

Look, do I think there’s value in looking at vertical leap, short-shuttle times, performance in college, and games played?

Sure. But I don’t think those are the be-all and end-all statistics that you seem to be touting them as.

by Requiem on Sep 4, 2011 5:48 PM CDT up reply actions

You seem to have a problem yourself with skepticism.

I understand what an adjusted r-squared of 0.42 means. and I know sackseer isn’t infallible.

I have a problem with skepticism? Right … that’s why I wrote this …

btw, if you want a couple counterpoints to the SackSEER methodolgy, Carlos Dunlap and Adam Carriker are examples where it didn’t work.

my problem is the majority of the criticisms are full of errors. like

Forster is a lawyer → lawyers are advocates → therefore forster is acting as an advocate not a scientist ???

I know that Nate Forester (the lawyer) developed it. I never said it was FO’s proprietary formula, did I?

???

I wonder, how valid this belief is, though, given:
The supporter of the stat is a paid site (footballoutsiders.com) which could benefit financially if they’re able to get the stat legitimized.
There’s no peer review of the work nor ANY WAY TO PEER REVIEW the work, as it’s a “proprietary formula”

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:02 PM CDT up reply actions

Again

I understand what an adjusted r-squared of 0.42 means. and I know sackseer isn’t infallible.

I have a problem with skepticism? Right … that’s why I wrote this …

Great. One example out of the many writing about Sackseer that you’ve done. I definitely don’t have a full measure of the amount that you’ve written about Sackseer, but from what I can recall, the vast majority of your time talking about Sackseer seems to support it. Which isn’t warranted for a regression, INCLUDING THE DATA USED, of 0.27.

And why have you not responded to that issue AT ALL? For that matter, why have you not brought up other potential issues you might have with Sackseer?

my problem is the majority of the criticisms are full of errors. like

Forster is a lawyer → lawyers are advocates → therefore forster is acting as an advocate not a scientist ???

They were not criticisms. They were reasons why I started suspecting that Sackseer may not be all they it is touted to be.

If you want to fundamentally say they are, well, I have no control over what you believe in. But I can tell you that’s not what I meant them to be.

by Requiem on Sep 4, 2011 6:19 PM CDT up reply actions

And why have you not responded to that issue AT ALL?

because I was going to email you because I didn’t feel like have a retarded back and forth on the board.

but tell me … do you agree with this:

What’s the regression? The dependent variable is sacks and the independent variables are SS, VJ, games missed, and SRAM right. So the string of data is

Sacks (player A) vs (SS, VJ, games missed, SRAM)

Except for later round players here’s what your input data is going to look like

Player A: 0 vs (SS, VJ, games missed, SRAM)
Player B: 0 vs (SS, VJ, games missed, SRAM)
Player C: 2 vs (SS, VJ, games missed, SRAM)
Player D: 0 vs (SS, VJ, games missed, SRAM)
Player E: 0 vs (SS, VJ, games missed, SRAM)
Player F: 0 vs (SS, VJ, games missed, SRAM)
Player G: 0 vs (SS, VJ, games missed, SRAM)
Player H: 5 vs (SS, VJ, games missed, SRAM)
Player I: 0 vs (SS, VJ, games missed, SRAM)
Player J: 0 vs (SS, VJ, games missed, SRAM)

Later round players mostly are going to be zeros. Zip nada. They aren’t going to get on the field and they aren’t going to be getting sacks. The majority of the sample is going to be white noise so of course the adjusted r-squared is going to be low. That’s probably why he ran the regression on rounds 1 & 2 in the first place, you actually have some results (i.e. sacks) to regress against. It’s impossible to regress against a string of zeros.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:26 PM CDT up reply actions

Right

Which is why touting the 7-round regression is butt-ass stupid.

He should just leave it out, but to me, it seems like he wanted something else to argue for the viability of his prediction. Specifically, he wanted to quell the potential data-dredging problems that smart people were going to bring up.

by Requiem on Sep 4, 2011 6:31 PM CDT up reply actions

it’s not a bad thing to anticipate and address criticism. of course he’d want to address data-mining. that’s the first thing people bring up.

comments on the new lewin forecast

http://www.footballoutsiders.com/stat-analysis/2011/introducing-lewin-career-forecast-v20

#3: How can I tell this system was created through over-use of regression

#5: I would really like to see some out-of-sample validation runs from this regression. It certainly has the appearance of a potentially over-parameterized regression

etc etc. I could go on and on … but that’s a common criticism.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:52 PM CDT up reply actions

Agreed that

it’s not a bad thing to anticipate and address criticism.

However, he goes about it in a completely stupid way, IMO.

Instead of citing something which has really no relevance, he should just flat out state:

Are there potential problems with data-mining? Yes. Will we be using out-of-sample data to try and verify how solid this predictor is? Yes.

He shouldn’t be using a bad sample to try and address the criticism. That just looks like he’s trying to cover things up.

by Requiem on Sep 4, 2011 6:55 PM CDT up reply actions

Also

I’d appreciate it if you’d refrain from pushing your suppositions on me and stating them as fact. Even if it is couched in a manner to give you plausible deniable, it’s still irritating.

I don’t KNOW that the sackseer predictions are wrong, but I’m definitely heavily skeptical that this will turn out to be worth more than the amount of time we spend discussing it.

You should compare and contrast to the amount of work done in the baseball community about BABIP, FIP, etc. Where there was a LOT of peer-review and talk and discussion about it.

This has nothing remotely like it, and it’s rather sad that this type of work seems to have garnered so many defenders without even having 1/1000 the amount of work that was put into BABIP.

by Requiem on Sep 4, 2011 6:01 PM CDT up reply actions

PCKB

I’d appreciate it if you’d refrain from pushing your suppositions on me and stating them as fact. Even if it is couched in a manner to give you plausible deniable, it’s still irritating.

You seem to have a problem yourself with skepticism.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:03 PM CDT up reply actions

No

You seem to have the attitude the you do know for certain … you know it’s wrong.

If I were to do what you do, I would have said:

You have a problem yourself with skepticism.

It may be nitpicky, but that only seems fair, since you nitpicked me.

by Requiem on Sep 4, 2011 6:09 PM CDT up reply actions

did you not see the ‘seem’ in the first half of that clause?

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:14 PM CDT up reply actions

I did

that’s why I said “couched in a manner to give you plausible deniable (sic)”

I would have had no problems if you had stated.

You seem to think that you know it’s certain that it’s wrong.

Saying perhaps the same thing, but the impact is different.

by Requiem on Sep 4, 2011 6:22 PM CDT up reply actions

I don’t know why you feel the need to make a bunch of your criticism an ad hominem attack on FO.

the criticism about the data mining is a reasonable and so is your skepticism the 7 round regression not demonstrating that it wasn’t data mined. But the rest of the stuff seems irrelevant.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:12 PM CDT up reply actions

Again

they weren’t criticisms.

They were reasons for why I started getting suspicious of this whole Sackseer phenomenon.

the criticism about the data mining is a reasonable and so is your skepticism the 7 round regression not demonstrating that it wasn’t data mined. But the rest of the stuff seems irrelevant.

And it’d be nice if you actually stated this, instead of just straight attacking my posts for what you perceived as criticisms.

That would lead people to believe that you weren’t just an advocate of Sackseer.

by Requiem on Sep 4, 2011 6:25 PM CDT up reply actions

Hmmm, it doesn’t look good for sackseer at this time. The lowest two predictions are currently doing the best out of the pass-rushers predicted for.

I disagree with your assessment. In terms of the relative ranking it looks poor, but in terms of warning teams off of bad players it’s doing exactly what it Forester says it does.

Like Playmaker Score (our measure for projecting college wide receivers), SackSEER is more accurate at identifying busts than it is at singling out potential stars.

do you think Philly is glad they used the 13th pick on a player who’s on track to have 15 sacks in his first 5 years? Ten is happy they used the 16th pick who’s on track to have 7.5 sacks in 5 years? NYG are happy they used the 15th pick on a player who’s on track to have 22.5 sacks in 5 years?

What Sackseer said was the 2010 was a weak class of pass rushers. And it looks pretty good in that regard.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 8:29 AM CDT reply actions

I disagree with your assessment. In terms of the relative ranking it looks poor, but in terms of warning teams off of bad players it’s doing exactly what it Forester says it does.

And it warned teams off of the two best performers in the group.

do you think Philly is glad they used the 13th pick on a player who’s on track to have 15 sacks in his first 5 years? Ten is happy they used the 16th pick who’s on track to have 7.5 sacks in 5 years? NYG are happy they used the 15th pick on a player who’s on track to have 22.5 sacks in 5 years?

What Sackseer said was the 2010 was a weak class of pass rushers. And it looks pretty good in that regard.

I’d wonder how 2010 fared wrt other draft classes, and whether the drop off was as steep. Also, I wonder what scouts themselves were saying about the 2010 class of pass-rushers. Like an auction, it only takes 1 team to like a player to draft them high.

Since Sackseer is the one professing the extraordinary ability, I think it falls on Sackseer and their advocates to defend their claims.

So, I leave it up to you to do the legwork to see if 2010 REALLY was supporting Sackseer ability or whether other people were saying the same thing and Sackseer didn’t really tell us much more than we already knew.

by Requiem on Sep 4, 2011 6:07 PM CDT up reply actions

I’d wonder how 2010 fared wrt other draft classes

well Forster was said 2010 was a weak class. it was the conclusion of the 2010 write up (before the season)

According to conventional wisdom, the 2010 edge
rusher class is strong and deep. However, SackSEER
believes that this year’s edge rusher class draft is not
particularly stellar at the top end. This class is plagued
by mostly poor shuttles, mediocre vertical leaps, and
no standout prospect in the productivity department.
If we combine the projections for the last two years,
Jerry Hughes would be the only 2010 draftee in the top
five. SackSEER rated Connor Barwin, Aaron Maybin,
Brian Orakpo, and Clay Matthews higher than the second-
ranked edge rusher in this class, Derrick Morgan.
Although some NFL team may indeed have found the
next Willie McGinest or Dwight Freeney, most teams
that drafted edge rushers early this year will instead
get a few years of mediocrity and the frustration of
unmet potential.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:22 PM CDT up reply actions

Again

I leave it to you to get a consensus on whether the draft class was considered strong or weak.

I don’t trust Forester enough to believe his take on what the consensus was.

by Requiem on Sep 4, 2011 6:33 PM CDT up reply actions

and btw, do you think the alternative is better? look at how sackseer does against the alternative: NFL scouts. Here’s the order players were drafted.

Brandon Graham 3
Jason Pierre-Paul 4.5
Derrick Morgan 1.5
Jerry Hughes 0
Sergio Kindle 0
Carlos Dunlap 9.5
Everson Griffen 0

would you say?

Hmmm, it doesn’t look good for ~~sackseer~~ nfl scouts at this time. The ~~lowest~~ 2nd lowest ~~two~~ prediction is currently doing the best out of the pass-rushers.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 8:42 AM CDT reply actions

Why does this have to be:

Sackseer vs. Scouts, which is better?

Why can’t it be:

Sackseer vs. Scouts, they both are extremely flawed.

by Requiem on Sep 4, 2011 6:26 PM CDT up reply actions

I’m cool with that. and you are too. and that would make 2 of on this board.

if you want to know why I’m heated it’s because 99% of this board isn’t gonna get any nuance. It either works / doesn’t work. I expect to have people posting how Requiem said Sackseer over-fitted data and my mom’s fat every time I mention Sackseer now.

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:33 PM CDT up reply actions

Well......

perhaps, you shouldn’t just post Sackseer data, then? :p….

I can understand your concern, but to be honest, I think it’s MORE damaging to support something which seems to have so many flaws (kindablue highlighted another issue with R-squared and adjusted R-square).

If I were looking at a draft guy, I’d probably be more interested in seeing more raw numbers, in general (including 10-20 yd times), instead of things specifically focused on Sackseer.

I dunno, maybe it’s because I come from the Ranger’s blog, but we have quite a few quality people there, in terms, of understanding statistics, and I’d like to think it’s because we try to describe the nuances in the debates.

But maybe it’s just because baseball CAN be more statistically driven than football can, since it’s more discrete and individualized…..

by Requiem on Sep 4, 2011 6:52 PM CDT up reply actions

send me an email

You know what Kenny Powers says? Fundamentals are the crutch of the talentless.

by Fan in Thick and Thin on Sep 4, 2011 6:52 PM CDT up reply actions

Why does it have to be Either/OR, Requiem?

Now I am not sure if you caught my draft series this past year, “How they would Fit”. In it I used Sack SEER + scouts(usually Wes Bunting) to compare and contrast the possible draft picks….

Now I am not saying that my way is the best, but wouldn’t it make sense to use every resource available?

the reason I use SackSeer numbers is due to this philosophy:
All things being equal, Similar measurables = Similar performance….

this of course is backed by a scout’s eye view of the player……

Here’s a theoretical play from 2010: Snap. Tony takes 7 step drop. Tony looks left at Miles, who is doubled, and looks right to where Roy Williams should be…but instead sees Colombo on his back and a Defensive End foaming at the mouth jumping over Marc’s carcass. Tony proceeds to run like hell and look for Witten
-by CotySaxman on Jul 11, 2011 7:50 AM PDT

Am I the most optimistic Cowboys fan in the World? Yes, due to an "unfortunate accident" to the previous holder of the title.

by I am Ironman!!! on Sep 4, 2011 8:40 PM CDT up reply actions

It's interesting you ask that

Baseball actually used to be in that situation about 20 years ago, when it combined scouting with some not very good statistics (BA, Rs, RBIs).

Now, it’s gotten to the point where you can actually make fairly accurate projections based on minor league numbers. So much so, that for the most part, you can measure and judge trades for prospects fairly decently when they occur.

However, before we could get to that level, we needed to improve on the statistics that we had (or junk them all together), to attain a level of knowledge that’s well in advance of what we had compared to 20 years ago.

IMO, this is the process of attaining any knowledge. And part of that is questioning any statistics and the methodology behind them to see if we can show flaws in them. This will lead to better knowledge and better statistics, in general.

Now, there will always be a human element to it. For example, in baseball, though a lot of statistic exist to evaluate a player, scouts are still needed to be able to identify those talents that will get there before their performance shows. Or to be able to identify the will to improve and work at it that leads to “overachieving.” Or for that matter to find the issues that stats and projections can’t see (e.g., holes in swing, instincts, how players handle adversity, etc.).

I sort of envision football getting to that stage also. But I’m not sure if it will ever get there. BUT to get even close to that, we need better statistics, and from what I’ve seen, sackseer doesn’t really provide much that the scouts don’t already cover, but I’m open to the fact that maybe it’ll tell us something interesting that we didn’t already know.

One thing that I’d be interested in if I were a football GM trying to be sabermetrically inclined, is to try and improve on our metrics here. One thing that I would like to see changed is instead of the combine being about players just running in their “underwear.” I’d want to measure players with the full set of pads on. That’s the more accurate measure, IMO. Even then, there are still quite a bit of changes that would need to occur before we’re close to the baseball level…..

And even then, we might not get there.

So, I guess to focus my meandering, my point is that Sackseer doesn’t seem to provide much more than scouts already do. But I’d be open to possibility that maybe there’s something there which it highlights. However, my issues with the methodology and lack of statistical chops leads me to be pretty skeptical of the process, in general….

by Requiem on Sep 5, 2011 2:46 AM CDT up reply actions

as far as Football getting to where Baseball is stat wise

I honestly don’t see that happening….the sample sizes in Football are just so much smaller than Baseball, plus football is more more team oriented, that it may be impossible to get the level of individual stats that you can get with Baseball…..

However….

my point is that Sackseer doesn’t seem to provide much more than scouts already do. But I’d be open to possibility that maybe there’s something there which it highlights.

Actually I think that the SackSeer guys failed to use the right metrics….in a sense….

The way I break it down is this-

Short Shuttle=Run stopping ability….By focusing on body control in a confined area, the 20 yard shuttle most resembles a defensive players ability to “Break down” at the point of attack vs a running play also correlates ability to “set up LIneman”…ie start outside then shoot inside gap….Also can be used to determine Coverage ability….

3-cone drill- “speed rush”- or ability to “turn the corner”

Vertical jump – useful for run/pass rush On the line DL….indicates explosiveness in run defense for LB’s and tackling ability

Broad Jump- Pash rush(first step) explosiveness for LB’s, also could be used for “Route jumping” for CB’s…..

so a player that has a good SS time but a “slower” 3-cone time would be a “run stopper” and a good 3-cone time coupled with a “slower” SS time would equal a situational pass rusher type….

The same with the “jumps”… a good Vert, combined with a poor Broad probably means that the player would be better off “on the Line”

where as a “poor Vert” and a good Broad means that the player might be better “off the Line”

Another thing that you have to look at is comparable “weight classes”.. ie say 250-260lbs, 260-270lbs, and so on….

a player that weighs 250lbs will more than likely “perform” better than a 300 lb player, so what you have to do is compare them to players that are of similar weight…

all this is just my opinion only, and is simply based off common sense and comparing the “drills” to actual Football plays…..does any of this make sense?

Am I the most optimistic Cowboys fan in the World? Yes, due to an "unfortunate accident" to the previous holder of the title.

by I am Ironman!!! on Sep 5, 2011 6:33 AM CDT up reply actions

I think the Giants are happy w/ JPP

He has talent.

The problem with Brandon Graham is that he got hurt. He lacks durability not talent.

One year is not enough to judge any player. Chief’s tackle Irv Eatman thought Reggie White was thought of as a big fat kid his first year in the USFL .

Only LT and Jevon Kearse really lit up the NFL their rookie seasons.

by Jonathan Stern on Sep 4, 2011 8:43 AM CDT reply actions

Much ado about nothing.

Interesting write-up, Requiem. Doesn’t look or sound like it’s intended to be a eulogy or euthanasia, despite the wailing. This “thing” looks to be what it was and is, an interesting perspective, an evaluated opinion, not a scientific analysis.

Cry 'Havoc!'

by tanstaafl on Sep 4, 2011 9:03 AM CDT reply actions

I thijnk Sackseer is useful

Not perfect but useful.

by Jonathan Stern on Sep 4, 2011 9:08 AM CDT reply actions

In defense of sackseer

Professional career
Pre-draft

Kearse, at 262 pounds, ran a 4.587 second 40-yard dash, 4.24 second 20-yard shuttle and had a vertical jump of 37 inches at the 1999 NFL Combine.8 Kearse lowered his 40-yard time to 4.43 at the University of Florida “pro day” in the spring of 1999.9

by Jonathan Stern on Sep 4, 2011 9:24 AM CDT reply actions

http://en.wikipedia.org/wiki/Jevon_Kearse

by Jonathan Stern on Sep 4, 2011 9:25 AM CDT reply actions

how in the hell jaso n pierre paul missed 26 games?

wow haha

by ratware on Sep 4, 2011 2:01 PM CDT reply actions

He missed them in college

by kindablue on Sep 4, 2011 2:42 PM CDT up reply actions

Specifically

he was a JuCo transfer.

And Sackseer heavily penalizes for JuCo play. If you were a JuCo transfer, you get no credit for that.

by Requiem on Sep 4, 2011 6:58 PM CDT up reply actions

Excellent article, Requiem

You’ve expressed a number of concerns I have with Football Outsiders’ approach to statistics. Good statisticians don’t relentlessly champion their ideas, they let the analysis speak for them. Nor do they withhold their analysis from peer review. By definition, a peer review means someone else could review the data and analysis and arrive at a similar conclusion. It’s a poor practice in the statistical community to not offer that analysis for review, and FO has been operating with so-called proprietary formulae for nearly a decade.

One thing worth mentioning is that r-squared is not nearly as relevant to a regression analysis as is adjusted r-squared. With four independent variables driving SackSEER, it will have a significantly penalized adjusted r-squared value. Not that 0.27 is anything to get excited about to begin with.

With that in mind, your evaluation of SackSEER may be too harsh, and might be suffering from selection bias. If the model looks at the sack totals over a five year period, your review of the youngest players on the spectrum could be unduly distorting the results, because many young defensive players don’t reach approach their peak performance in their first or second year. I think to be fair we have to evaluate the model over the intended period of five years to judge it.

But the above reservation aside, it’s always refreshing to come across someone who clearly understands the fundamentals of statistics. It’s a field that is easily misunderstood and oftentimes abused—as you implied in your article.

by kindablue on Sep 4, 2011 2:40 PM CDT reply actions

Thanks for the insight

I’m don’t consider myself much of a statistician, to be honest. But I feel that I have more chops than Forester does, which instantly makes me skeptical of his work.

Though I agree about what you said about FO, I do find a lot of their work enlightening, or at least, thought-provoking.

I guess part of why I’m not as embracing of the football stats community is because my exposure to the baseball stats community and the freedom of peer review in that community, and how truly interested guys are in getting the right answer instead of advocating for their pet projects.

The football stats community seems closed and more like religious fanatics than the baseball stats community (though we’ve got those there too :p).

I agree with you about the evaluation being too early. Maybe I should have labeled the article, “A Too Early Evaluation.” I was just curious how it was doing since it was brought up.

Also, thanks for highlighting the adjusted r-squared vs. r-squared issue. Didn’t even think about that (but then again, as I said, I don’t consider myself much of a statistician since I don’t have to deal with it much in my day job).

by Requiem on Sep 4, 2011 6:44 PM CDT up reply actions

Ironically

The foundation of most statistics is just common sense. Too often people confuse the profession of statistics with obsessive number crunching by shamans, when in fact the most important thing is to properly orient oneself to the fundamentals. To your credit, I think you have that down pretty well.

I also agree that football opinions, including most regarding statistical analysis, tend to be more akin to religious beliefs rather than open-minded intellectual inquiry. As a result, the analysis of football statistics is sadly stuck in the Dark Ages, similar to baseball in the early years of research by Bill James. Perhaps because football is more of a violent, emotional sport than baseball, that character can cloud people’s minds, although that just speculation on my part.

Football Outsiders has done some good things, I agree. I like their receiver catch rates and some of the offensive line metrics they’ve created. Unfortunately, their cornerstone analysis, DVOA, is so flawed that it has no practical meaning. And they can be awfully defensive when discussing their shortcomings, and that lack of candor and maturity does not flatter their analysis.

by kindablue on Sep 4, 2011 7:07 PM CDT up reply actions

Agreed

I do wonder, though, “How much can we actually quantify with football?”

I have some ideas of what I’d like to potentially measure, but currently none of that information is really kept.

For example, one important statistic that would be nice to measure how long an OL provides the QB time before pressure gets to him for each play. Then, you can REALLY get a sense of how good an OL is versus other OL. Or another example, is how many times a wide receiver actually gets open vs. a defense.

But we still run into the wall of subjectivity (at least as an outside observer). For example, on the McMichael TD, who was really at fault for the blown coverage (Spencer or Ball)? That’s not something that is easily answerable unless you have a knowledge of what the assignments were. And THAT is not easily attainable, as DCs won’t want to be sharing what actual assignments there were for fear of other teams using that information.

So, I guess, part of the issue I have is, can we actually attain the same depth of knowledge in football that we have even in baseball? And even if we can, how much work do we have to do to actually obtain that knowledge?

by Requiem on Sep 5, 2011 2:56 AM CDT up reply actions

Short answer

Is there’s more there than you may realize. There is less in the way of individual statistics than in baseball, but you still can make very useful inferences from the data. Take the offensive line for example. It will be very hard to judge how good individual offensive linemen are by virtue of not knowing for sure who missed an assignment when a defender is unblocked. There’s also the issue of who is the competition; not all defenders are created equal. The bigger and in my mind more important question is, does it really matter? When a certain offensive line is playing, you know how well they run the ball and using my points per pass metric, you know how effective they are when they pass it. And in the end, that’s probably all you’ll need.

The reservation I have with diving into the minutiae of individual performances is two-fold. One, there is seldom a correlation back to whether that individual performance information affects the amount of points scored, which is the end result we should all be trying to measure. Second, given the deeply interactive and integrated team nature of the sport, I question in most cases whether individual performance can be meaningfully separated out. In short, I believe worrying about individual performances, especially at positions for which there are few statistics, is not a very productive venture.

There’s different ways in the world of statistics of arriving at meaningful conclusions. Sports fans often assume it has to be done by building things up from an individual level, but there’s ways around that. Parametrics and analogies are two common means, and have the advantage of not losing sight of the big picture.

by kindablue on Sep 5, 2011 1:42 PM CDT up reply actions