Grading the Graders: A Retrospective Review Of 2007 NFL Draft Grades

An old adage, one that pops up in particular during the post-draft "grading" season, suggests one cant judge a draft until several years later. In fact, many of the draftniks who dole out grades immediately (and I mean immediately) after a draft refer to this, often somewhat apologetically, even as they spoon out their grades. Certainly, these grades are fun and create a stir, fodder for further argument, etc. And, in recent years, I have seen several scribes go back and "re-grade" a draft (or even offer a new mock draft based on who should have been selected at each pick.

A bit more useful is the project engaged in by the fine folks at Football Outsiders. For each of the drafts since 2004, they have offered up a "Report Card Report," in which they review the grades each team received from the top draft pundits. The motivating force behind this exercise is to determine which teams' drafts achieved an agreed-upon consensus and which were the most divisive. While I don't find this aspect of the project tremendously useful, I find the data they compile to be of great benefit for the questions I'm interested in asking: can we establish a correlation between these consensus grades and teams' actual drafts? Which predictors tend to the be the most or least accurate? Which teams drafts were more accurately graded, and why?

Because we can only determine the accuracy of their grades with some historical distance, I'm going to look at the 2007 draft. The players selected in '07 have given us sufficient opportunity--four seasons--to judge their careers. Recall Bill Parcells' claim that you know about a guy after three years in the league; a superb example of this is Doug Free who, midway through his third year, looked to be a marginal player. When Marc Colombo was injured against Green Bay, however, his career took off. Being a generous spirit, I've given all the 2007 draftees an extra year to prove themselves--a Parcells plus one, so to speak.  My larger aim is to take a harder look at the 2007 draft, in the hopes of answering some of the above questions.

The method of my madness and some results after the jump...

The first step is to assemble all the various grades doled out by the sportswriters in question. I've found draft report cards from Mel Kiper, USA Today, Sports Illustrated's Paul Zimmerman and Jeff Chadiha, CSBSports' Pete Prisco. Walter Football, Football's Future, and  consensus grade compiled by Football Outsiders and The Feed (sadly, I couldn't find a link to Goose Gosselin's grades). Using so many different and varied graders introduces a statistical quandary: not all of them use the same scale. As FO writer Jake Schumacher puts it:

these are sportswriters we're dealing with, so we'll need to take a few steps if we're going to make any sense out of things. First, remember that they actually rate the average draft as above average, about a B-....Also note that Dallas Morning News' Rick Gosselin grades the toughest, about a C+ on average, and Sporting News' War Room Scouts is by far the easiest on teams, with a B/B+ average...[and Walter Football skews things by giving out A+ grades, so]...I rate agreement using a simple statistical measure of spread, standard deviation.

In the table below, you can see the adjusted grades handed out by Shumacher (the column marked "F.O.") alongside a non-adjusted average of all the other grades (column marked "Avg.") In some cases, this correlates quite closely; for example, Football Outsiders gives Atlanta a 3.4, while the average grade is roughly the same: 3.47. On the other hand, several grades are non-correlative; the most extreme example is Denver, where a 2.3 from the FO guys is contrasted by a 2.72 GPA from the assembled Internet pundits. Check out the entire chart:

Thus far, this is little more than a fun exercise--a pleasant comparison of various asseverations (some statistically enhanced) made four or more years ago. What I'd like to figure out here is how accurately these drafts were graded, and why. To do that, I must first go back and determine the success--or lack thereof--of each team's 2007 draft. To do this, I turned to my friends at Pro Football Reference. Using their matrix for "Weighted Career Approximate Value," I determined how well each team drafted. I assumed that a team needs three of its draft picks to become regular contributors for a draft to receive a passing grade. I designated as a contributor any player with a CAV of 10 or higher.

From this, I developed the following rubric:

A: five or more contributors; four contributors + star (CAV of 30+) or superstar (CAV of 45+)
B: four contributors; three contributors + star or superstar
C: three contributors; two contributors + star or superstar
D: two contributors; one contributor + star or superstar
F: one contributor

Looking at the chart, you can see the grades derived from this rubric in the far right column (marked "PFR"). In comparing them to the grades given after the draft, there is actually surprisingly little deviation in the grading accuracy of our assembled draft gurus. In a bit of an upset, the most accurate (within a third of a grade) were the guys at Football's Future, followed closely by Prisco, with 14 accurate grades. All the rest were on the mark eleven times.

What can we learn from this? Teams without first-round grades tend to receive bad grades. In 2007, neither Seattle nor Philadelphia had a first round selection; as the chart demonstrates, they received the two poorest average grades from our collective draft evaluators. As it turns out, however, both had poor drafts, although I suspect that this is not because they didn't have first round picks--Green Bay, after all, botched their first rounder and then went on to have arguably the strongest draft that year.

Perhaps most surprising is that, although individual grades aren't particularly impressive in their accuracy, the composite draft grades are actually quite accurate (even for the Seahawks and Eagles), and end up correlating much more closely that I would have expected to the actual performance of 2007's rookie class.

The ones that don't are of particular interest to me; these fall into three categories:

1. The team in question had an unpredictably deep and productive draft, particularly in the late and middle rounds. Both the Giants and Packers hit on players throughout and ended up with multiple contributors. The Giants were dinged by draft graders for failing to address needs, principally on the offensive line. Green Bay was marked down for first round choice Justin Harrell--rightly so; his CAV sits at a puny 2--but hit on several other mid-rounders.It was not by accident that these teams that met in the 2007 NFC Championship game.

2. The team had an unpredictably bad draft. Denver and New Orleans each had two low-level contributors; San Diego, New England and Chicago ended up with one each. The Patriots traded their second rounder for a first in 2008, and their fourth rounder for disgruntled Raider wideout Randy Moss. The others? Epic fails, all. One problem is that very few graders are willing to dole out low grades; there are no "F"s and only a smattering of "D"s.

I don't think there's any way to account for either of the above occurrences--I'm no statistician, but I'd bet that the number of unexpectedly good and unaccountably bad drafts falls along some kind of standard deviation, especially if we account for the fact that most draft pundits grades tend to cluster high (above an average "C"). There is an interesting third category here, however:

3. The evaluator was off the mark because he overrated quarterbacks. A look at the first 50 picks of the draft shows 11 players with CAVs below 10. Of those, five are quarterbacks. Yet most of these teams were lauded after the draft for shrewdly picking up their quarterback of the future. In particular, Cleveland became draft day darlings (they had Football Outsiders highest average grade) by trading up with Dallas to nab Notre Dame signal caller Brady Quinn, who has since taken his CAV of 3 to Denver.

I'll continue to do this every year to see whether my observations about the 2007 draft hold true for other years. And, when draft grades come out a year form now, I'll average them out to determine whether our brave 'Boys have netted a good haul--that is, unless they draft a quarterback or trade their first-round pick.