When Analytics Fails

While analytics continues to reshape the NBA, the irony is it’s creating more questions than answers. NBA teams want black and white, but increasingly, gray is the result. The holy grail of objective player analysis is replaced by the reality of more and more subjective opinions, often overconfidently backed by an array of statistics. Pity the thankless job of a NBA General Manager, tasked with separating the barely audible signal from the ever-deafening noise. The challenges are obvious: in an endless sea of advanced metrics, which ones are actually meaningful and additionally, how should they be weighed against all those visible yet unquantifiable traits; a player’s so-called “intangibles”? Welcome to the never-ending analytics vs. eye-test debate.

Unfortunately, as the debate rages on, more fervent now, than ever, the arguments, as they’re currently constructed, miss the most prominent point. That is, the shortcomings of the human mind swirling with cognitive biases. They cloud even the smartest NBA executive’s judgement. The scariest part is that these biases lurk in the shadows, operating in the deepest recesses of the brain, shaping conscious decisions through unconscious means. Analytics tries to solve this problem by removing the human mind from the decision-making process, but the catch-22 is that the biases themselves are inescapable. No matter the context, analytics or otherwise, the human mind is fully capable of painting fallible pictures with very little regard for accuracy.

It’s no surprise that this conundrum has caught the attention of Houston Rockets GM, Daryl Morey. After all, the Rockets are widely known as one of the most forward-thinking NBA teams, unafraid to test ideas at the boundaries if they believe even a small advantage is possible.

Morey’s first legitimate exposure to the human mind’s vulnerabilities came after enrolling in a behavioral economics course at Harvard Business School during the 2011 NBA lockout. The story is as follows:

At the start of the first class, the professor asked him and everyone else in the class to write down the last two digits of their cell phone on a sheet of paper. Then she asked the class to write down their best estimate of the number of African countries in the United Nations. Then she collected all the papers and showed them that the people whose cell phone numbers were higher offered systematically higher estimates of African countries in the United Nations. Then she took another example and said, ‘I’m going to do it again. I’m about to anchor you. Here. See if you aren’t screwed up.’ Everyone had been warned; everyone’s minds remained screwed up. Simply knowing about a bias wasn’t sufficient to overcome it.

The thought of that made Daryl Morey uneasy.

Of course, Morey had experienced cognitive biases driving poor decision-making prior to the behavioral economics class at Harvard. Most notably, passing on current Memphis Grizzlies center, Marc Gasol, in the 2007 draft.

At the time, after a not so flattering shirtless photograph of Gasol surfaced among Houston’s scouts, the staff had nicknamed him “Man Boobs.” In turn, Morey let his convictions about Gasol’s likely NBA success go by the wayside. As Morey recalls, “That was my first draft in charge and I wasn’t so brave.” The Lakers ended up taking Gasol 48th overall. To date, he’s one of only four All-Stars in that draft class and ranks 3rd in win shares behind only Kevin Durant (2nd pick) and Al Horford (3rd pick). In hindsight, the “Man Boobs” label clearly led to a devaluation of Gasol’s basketball talents within the Rockets’ ranks. Morey recognized this. “’I made a new rule right then,’ said Morey. ‘I banned nicknames.’”

Morey also speaks regretfully about the 2008 draft. Houston originally drafted Nicolas Batum, but ended up swapping him for Joey Dorsey in a 3-team trade involving Portland and Memphis. Not only that, Houston missed out on eventual 35th pick, DeAndre Jordan. Dorsey’s career was short lived, accumulating just 2.9 career win shares. Meanwhile Batum (46.9 career win shares) and Jordan (66.9 career win shares) are still going strong. Morey attributes the mistake to Dorsey’s likeableness during his interview and the unknowns surrounding Batum (a youngster from France) and Jordan (one half-hearted season at Texas A&M) prior to the draft.

In Dorsey’s case, it was a prime example of the halo effect, the tendency for an impression created in one area (e.g., personality in an interview) to influence opinion in another area (e.g., basketball skills). And as for trading Batum and passing on Jordan? The cognitive bias known as the ambiguity effect led Morey and the Rockets to avoid two players for which missing information made the probability of NBA success seem “unknown.”

Over the years, Morey has identified plenty of other leaps made by the human mind within NBA circles, Rockets included. For instance, confirmation bias, which Morey calls “the most insidious” is the tendency to interpret new evidence as confirmation of one’s existing beliefs or theories. Nothing like forming an opinion of a player and then seeking out evidence to back it up.

“’The classic thing,’ said Morey, “and this happens all the time with guys: If you don’t like a prospect, you say he has no position. If you like him, you say he’s multipositional. If you like a player, you compare his body to someone good. If you don’t like him, you compare him to someone who sucks.’”

Clearly, the inability to separate prejudices from reality leads to repeated failures among NBA talent evaluators. And if the job wasn’t already difficult, players themselves look to capitalize by drawing their own favorable comparisons.

For example, Morey believes there’s no way Eastern Washington University product, Tyler Harvey, even gets drafted in 2015 without the existence of Stephen Curry. The similarities are striking: two light-skinned, mixed race, slight of build, point guards who lit it up from deep at relatively unknown universities. For his part, Harvey made the rounds during pre-draft interviews citing Curry as the player he most resembled. The Orlando Magic bit, nabbing him with the 51st pick. Unfortunately, Harvey has struggled mightily in both his lone G-League season and a stint overseas in the Italian League’s Serie A. Odds are he’ll never see NBA minutes, a far cry from the four-time All-Star, two-time MVP and two-time NBA Champion he likened himself to.

In 2015, the Rockets, as always, held more pre-draft interviews. One particular interviewee went by the name of Satnam Singh. At 7’2”, 285 pounds, the Rockets were reminded of their own former Hall of Fame center, Yao Ming. Singh spoke at length about his incredible and well-documented (see ESPN The Magazine and Netflix) journey from his 800-person Indian village to the NBA’s doorstep. Almost on cue, Singh’s own comparison to Shaquille O’Neal came and went. And then after all was said and done, and Singh had exited the room, Morey asked the looming question, “Have we found evidence he has played organized basketball anywhere?” The short answer was “No.” With no data on Singh, Morey would have to pass and pray Singh wasn’t the second coming of DeAndre Jordan and another missed opportunity traced back to the ambiguity effect.

Although he’s yet to play a NBA minute, the jury is still out on Singh, who, oddly enough, was selected immediately after Tyler Harvey in the 2015 draft with the 52nd pick by the Dallas Mavericks. And as for people’s awareness (or lack thereof) of the mental processes that even analytics can never solve, Morey sums it up pretty well, “It was like a fish not knowing he is breathing water unless someone points it out.” (The Undoing Project, 2017)

One thing is for certain, the NBA analytics vs. eye-test debate may never be settled. To be clear, the conversation is a worthwhile one, just as long as we continually incorporate the vagaries of the human mind. Seemingly, the more we learn, the less we know. This is a lesson that applies to both basketball and life.

The classical Greek philosopher, Socrates, may have said it best, “I know that I know nothing.” Remember this the next time you go to make that unproven player comparison or seek evidence for a preconceived basketball belief. Ultimately, awareness is everything.


photo via llananba

One thought on “When Analytics Fails

Comments are closed.