What if I came to you and said "I'm an expert predictor! Give me a 24 win range and I can tell you how well every team in the NBA will do?" Let me give you a practical example. I'm a Nuggets fan. Using the expert prediction skill I mentioned above, I will now make the following prediction:
Next season the Nuggets will win between 40 and 64 games!
Useful right? For the last several seasons, wilQ at Weakside Awareness has done a checkin on how analysts do in terms of predictions at the start of the season and how the season plays out. And the way this is typically interpretted is "Wow John Hollinger is great at predictions! He beat Vegas!" But there's a key flaw here. You see, the story isn't Hollinger is good at predictions, it's that everyone is bad!
The metric wilQ uses is standard error. Now, I trust that wilQ knows the math, he's a very sharp analyst. However, I don't buy that everyone else does. A better way to think of standard error is "What's the range of games I need to place all of my predictions?" The way it works is:
I need twice my standard error to properly predict 20 teams. I need four times my standard error to properly predict 29 teams. With that in mind. Let's redo wilQ's chart of how our analysts did.
|Analyst||Range of games to get 29 teams "right"|
|Las Vegas Hilton||27|
|George J Monroy||28|
|TeamRankings Projections (BETA)||32|
|ESPN The Magazine and Basketball Prospectus||38|
|Wins in previous season||39|
Updating vs. Predicting
What's key about this is to stop using predictions as the end all be all test. Hollinger is another great example. In the middle of the season Hollinger went to the Memphis front office. And the Grizzlies, who are now in the conference finals, made several moves to their teams. Of course, Hollinger could not have known about these to start the season when he made his predictions. And yet, these moves had some influence on the Grizzlies' success. Using the best data at the time and updating our "predictions" as things progress is the right method. And that's why using metrics that have the best combination of
- Telling us what happened.
- Being consistent over time.
Are useful. We can use them to guage what is happening and use them to update our analysis about teams. Trying to become "perfect" at predicting just won't happen. Hollinger has "beaten" Vegas three years now. He hasn't improved the range of games he needs to do it though. And, as I mentioned, that range is rather large. The number of moving parts in a season is vast. It's good to give judgements about the state we think teams are in. Being able to accurately guage how they'll do after 82 games when injuries, trades, breakouts, breakdowns, etc. occur though? It's simply not going to happen.