Is Tom Izzo really the best tournament coach ever? Well...maybe. A look at subjective value assessments in real world analysis.

Statistical analysis is a tricky thing. The most common misconception (in my opinion) is that real world analysis is similar to an academic math problem. Most pre-graduate math problems have a clear structure, pre-defined assumptions, and most importantly, only ONE answer (makes grading easy of course). That reflects poorly on pre-graduate level math classes but that is another topic.

Coming from this background, it is easy to consider any piece of numerical analysis as a math problem that the analyst worked out behind the scenes and is now telling you the answer. In fact, this perception is commonly (and amorally) cultivated by the analysts themselves. This is a problem even when the analysis is done well. Real life problems almost always require layers of assumptions and perspectives to give the best possible answer. Subjective value assessments are unavoidably an integral part of analysis.

Take this article from fivethirtyeight.com which attempts to grade college basketball coaches on their NCAA tournament performance. First off this is a good piece of analysis (Ha! Everyone who knows I am a Michigan fan thought I was going to trash this article for lifting up Izzo as the best tournament coach). I do, however, take issue with the fact that the article claims a conclusion to a very complex and difficult to quantify matter (coaching ability) based on this one single metric, wins above expectation (WAE). Even a good metric needs context and alternate perspectives.

Taking a look at his data, it is readily apparent that the author simply added up every coaches' WAE across their entire career. The exceedingly obvious next step (not done in the article) is to divide that number by the number of tournaments each coach has participated in, thus producing a WAE per tournament appearance. For example, Tom Izzo has a career 14.6 WAE and 18 appearances, producing a WAE per appearance of 0.81. I grabbed a few coaches (not randomly, for the purpose of meaningful comparisons) from that list and calculated their WAE per appearance for context:

Brad Stevens: 1.62

Tom Izzo: 0.81

John Beilein: 0.77

Mike Krzyzewski: 0.14

Look! Tom Izzo is still really good. For those of you convinced I am working to undermine that conclusion you are going to be disappointed. I chose these coaches to make a few points. The number of tournament appearances matters. Tom Izzo and John Beilein have a similar WAE per appearance, but Tom Izzo has double (18 versus 9) the appearances. Assuming Beilein coaches many more years (please please please let this happen) and makes 18 tournaments, it is entirely reasonable to expect that he ends up with a similar WAE as Tom Izzo since he is already on pace to do so.

Another comparison is Mike Krzyzewski, who is a clear notch below the others on this list. The difference here is that he has four titles titles compared to one for Izzo and zero for the others. This sort of analysis treats all wins the same even though you could very easily conclude that a coach who wins it all one year then bombs out early the next year is better than a coach who hits expectation exactly two years in a row, depending on your perspective. Another factor that I won't go into too much detail here is that I think this metric is unfairly punitive to the higher seeds. Let's just put it this way: this metric determines that the average five seed winning two games (0.9 WAE) is a better coaching performance than the average one seed going to the final four (0.7 WAE). I think there is some gray area around that determination.

My final illustration is Brad Stevens, who in five tourney appearances with Butler produced 8.1 WAE, good for 1.62 WAE per appearance. When looking at WAE on a per appearance basis, it is Brad Stevens rather than Izzo who is the best tourney coach (-by far, as if that qualification was necessary in the original article. We have clearly shown that it isn't that simple). The illustration is not necessarily that Stevens is better than Izzo, but that it is difficult to determine the value of Izzo's long term consistently high performance versus the extraordinary short term performance of Stevens. It is a subjective comparison with no clear right answer.

And this brings me to the point I wanted to make all along. It is exceedingly tempting to eliminate the messy subjective comparisons by making your own determination and simply presenting your conclusion as if it is the "right" answer. I mean, when you do that you get to write click worthy titles like, "Izzo is the best tourney coach ever - by far" instead of the more accurate "Izzo is either the best ever or just one of the best ever depending on how you value his long term consistency versus Brad Stevens' remarkable five year run." And yet, the second is a better answer than the first.

As an analyst you want your work to be noticed and appreciated. Nuanced analysis tends to be ignored because most people don't want to take the time to understand all the underlying factors. This leads to either shoddy work (not the case of the above article in my opinion) or to the analyst making the subjective value determinations for you without telling you that he has done so (bingo). I think the author of the article in question simply decided that Izzo performing at a high level over 18 appearances made him a better coach than Stevens, who was only in five tournaments and would most likely regress to the mean in future years. Instead of telling you he did that, though, he just avoided the matter entirely. 

I think it is important for 1) analysts to be aware of this temptation and try their best to present the issue at hand in as fair and complete way as possible and 2) for consumers to always try to dig beneath the surface and make sure you agree with the assumptions in place before accepting the conclusions being presented.