False Alarms: Why We Say “No” When We Should Say “Yes”
Fear
Within numeric wisdom, as we seek to be enlightened,
Lurks circumstance and outcome where one soul will soon be frightened.
And on the field of battle, this poor schmuck will be exhorted,
“The data makes it clear” we cry, and leave them unsupported.
But strategies from leaders, be they CEOs or pharaohs,
Require more than models, but who fields those slings and arrows.
Amidst the idealized notions of a green grass, sunflower seeds, bubble gum, and hot dogs is the reality of little league baseball, as experienced by an anxious child. No, I wasn’t the kid in right field praying that the ball would never find him. That kid is waiting for the orange slices, the snow cone, and the freedom from the pastime he never wished to experience. I was something else entirely, a perfectionist. The diminutive only child, nerdy, studious, and willing to spend hours firing a tennis ball against the wooden paneling of a partially-finished basement for hours on end to refine eye-hand coordination. The coach’s kid who received a steady diet of ground balls in fading sunlight at the local park. Throw to the proper base. “The best throw is no throw.” I loved baseball. I hated the possibility of failure. The thought of striking out struck terror into my soul. Most would have navigated this anxiety by choosing something else to do on sunny spring days.
But not me. Baseball is everything a numerically-inclined, bookish kid could ever hope for. A rich array of statistics-driven narratives spanning a century penning new chapters annually. The trips to the stadium with father and its mixture of ritual and duty in carefully filling the scorecard with the details of every pitch. Baseball was a romantic pursuit. Catching a ball was the gateway to social acceptance as the new kid at school. And still, striking out was horrifying. How could I avoid the shame?
The answer was a shortened swing coupled with an almost pathological fixation with putting a ball in play when a pitch entered the strike zone. Though anxiety produces the type of plate discipline that generates a fair number of walks when opposing pitchers are navigating the awkwardness of their adolescent growth sports, I also generated more harmless ground balls than Mario Mendoza. I avoided a strikeout for literally years. In professional settings, we often cannot choose where and when we enter the batter’s box. The disapprobation of striking out is an unmistakable, career-damaging outcome. However, in far too many cases, the cost of risk-aversion is larger, more difficult to quantify, and a corrosive toxin that prevents taking the swing upside demands.
Data scientists will, without fail, perform the salient expected value calculation and estimate the probabilities of successes and failures acceptably. What neither they, nor the vibe-coding platforms that enable them can account for is the stage on which those failures occur. The world of professional tennis is filled with men and women who deliver their first serves with triple-digit velocity and pinpoint precision. These missiles land within the lines over 70% of the time for some elite servers and in excess 75% of the time for a few folks on the men’s tour with heights that wouldn’t be out of place on an NBA court. When these first serves succeed, the player’s probability of winning the point is extremely high. Alas, tennis offers servers only one opportunity to “miss,” after which the dreaded “second-serve” must be offered, lest that server gift their opponent with a free point. In their hopes of being spared the indignity of double faults, second serves are meeker attempts and the probability of winning the point behind the lesser offering falls significantly.
One particular professional of impressive stature and exemplary talent is Alexander Zverev. Standing 6’6’’, his first serve is formidable. His second serve, and the robustness of his psychological temperament have been questioned throughout his career. Famously1 he allowed a 2-0 lead in a best-of-five US Open final to evaporate, failing to complete his victory as he served for the fifth set.2
During 2020, Zverev landed his first serve at a rate of 68.3%. He won 77.0% of those points. When he was forced to deliver his second (weaker) offering, he won 45.0% of those points3. Worse, not all second serves, despite their comparably conservative approach are successful, and he double-faulted 6.4% of the time. This begets some trivial math, with which your handy excel spreadsheet or your friendly LLM companion will gladly assist you.
\[ P(\text{win on first serve}) = 68.3\% \times 77.0\% = 52.6\% \]
\[ P(\text{win on second serve}) = (1 - 6.4\%) \times 45\% = 42.1\% \]
Does everyone see the problem? He’d be better off hitting his first serve again! Even though he’d miss nearly a third of those attempts, he’d win the significant majority of the points in which that second (first) serve landed within the box. Moreover, were he to practice only his first serve, one would imagine both parameters in the first equation would improve.
This math is both incontrovertible and extraordinarily simple. Literally, multiplication is sufficient to illustrate the point. And yet. Omitted from this calculation and any other such depiction emerging from an AI is the particular flavor of human suffering the optimal strategy begets. Where, in the trivial mathematical expression lies the emotional wound of standing on the hallowed grounds of Wimbledon, looking up at the royal box with princess Kate and Roger Federer looking on…and striking dozens of double-faults as an overly-polite crowd of British patrons gasps?
Do you imagine this 6’6’’ racquet-wielding virtuoso is similarly adept at navigating the impending press-conference in which his stunningly contrarian strategy will be questioned relentlessly? Is he prepared to field inquiries about whether he will repeat that paradigm in all subsequent matches? Would he, in that fateful final, amidst the throngs of New Yorkers, dare to incur not the potential of defeat, but of a spectacle in which he plays a dubious starring role?
So, dear C{x}O, are you offering not only the data and quantitative expertise required for discovery of the optimal solution, but also the support structures required for those who will execute those plans?
Fear 2.0
When wondering why when adopting AI,
The discourse is crawling with doubters,
Remember the fate of tools driven by data
In novel, peculiar encounters,
There rarely is terror of dull, human error,
We know, all too well, imperfections,
But wand’ring the shallows of optimized algos,
We cringe at their errant selections.
One form of fear lies in our insecurities and our internal fragility. The last essay spoke of the reluctant little leaguer, afraid of the walk back to the dugout following a strikeout or the accomplished professional tennis player who would bear the slings and arrows of the crowd and future podcast punditry as the double-faults accrued. But now, in the age of AI, a second form of insidious concern strangles would-be upside in its crib.
Human beings are profoundly fallible creatures, plagued by their distractibility4, sloth, irritability, irrationality, and general stupidity5. However, we accept these limitations, and hand them the keys to two-ton hunks of steel filled with combustible fuel, and send them off to speed through densely populated urban centers. Predictably, from time to time, this goes horribly wrong. We recognize that the personal agency associated with driving embeds risk and costs for the individual and other passengers and pedestrians.
Because humans are permitted to err.
We allow human beings to move markets with the click of a button, transacting in dizzying sums at the risk of failures of cognition or dexterity.6 Sure, this might catalyze a capital-destroying sequence of reactions, but still place the sack of organic material before the keyboard.
Because human error is acceptable.
But if a driverless vehicle causes a fatality, the reaction to a (sadly) familiar outcome is wholly distinct. If an automated trading algorithm sparks a flash crash, the recriminations are of a different tone. Even if a self-driving car can demonstrate, over hundreds of thousands of miles, that its collision statistics are superior to human drivers, when the artificial intelligence produces a failure a human would have avoided, the discussion is rarely about aggregate and probability distributions. AI’s flavor of “error” often seems silly in the eyes of a “wiser” human being, especially if it fails to discern how to navigate a scenario human beings would consider trivial and obvious.
Among my avocational pursuits is the development of algorithms for the purposes of wagering upon sporting events. Humans who follow their gut transfer massive sums into the coffers of casinos. Algorithms (sometimes) deliver better outcomes. And yet, even an experienced algorithm leaves its operator to shudder and shake his or her head in disbelief at its failure. Perhaps it looks to back a starting quarterback who exited his most recent start with a gimpy ankle and a possibly-concussed cranium. Perhaps it fails to recognize that a major league baseball game played at an elevation in Mexico typically reserved for alpine hiking is likely to produce runs in quantities it lacks the historical comparisons to predict.
Should we eschew the silicon-based for the carbon-based? Of course not!7 Should we continue to allow hunks of gasoline-fueled vehicles to be steered by organic creatures fueled by gas station snacks and caffeine?8
Thus, when the boardroom discourse raises concerns about some AI-driven alternative, is the bearish narrative driven by anecdotal examples of errors a human would not make (if so, remember that self-driving cars don’t drive drunk or text at the wheel)? Or is the argument grounded in quantitatively-defensible reasoning and apt comparisons?
Only one is a reason to say “no.”
Fear and Fear 2.0 are drafted; the remaining sections of this chapter are forthcoming.
The Wrong Winners & Losers
Around the felted table, sit the experts and beginners.
The multitudes of miscreants, the myriad of sinners.
The flashing screens, fluorescent lights, and upper-crusty-dinners
The house, its edge, the game, the dice, the losers, and the winners.
Psychology, adversity, a tapestry of yearnings,
But in our game, ’tis not the odds, but who receives the earnings.
Most decisions fail to adhere to the clean definitions found in casinos. A corporate environment is not the floor of the Bellagio. Arguably, better decisions would be made if one particular feature of the modern casino was better replicated in the modern professional milieu. Sure, in the casino, the odds are stacked against you and every clock-free, consistently-lit, oxygen-infused component of the structure is meticulously designed to entice you to continue to play games where the odds are decidedly not in your favor9. But they possess one salient feature that incentivizes rationality - the winner keeps their chips.
This seems like a trivial observation. Of course we can keep our winnings and walk out the door, given sufficient discipline. Of course, since it is our hard-earned capital imperiled with every wager, surely10 we capture the upside in the case of favorable outcomes. How could it ever be otherwise? Who would participate in such a nonsensical charade?
Almost every corporate manager.
Each project, to the manager responsible for its outcome, is a hand at the poker table. If the project is a success, perhaps that individual will receive the vocational baubles of raises and promotions. In the case of failure, one risks terminations, layoffs, and other destruction of professional capital. So let’s consider a simple, contrived example11.
A high-risk, high-reward project that should require roughly one year of a team’s labor is to be discussed in some distributed video conference. The probability of its success is 25%. The ROI, should this project succeed, will be 10x. For simplicity, if it fails, the $1M is lost entirely. Bet $1M for a 25% chance of winning $10M. This is a trivial wager to consider if $1M is small relative to the annual revenue of the firm, and exceedingly trivial if the firm can undertake dozens of these high-upside projects in any given year…at least in terms of simple mathematics:
\[ \operatorname{E}[\text{profit}] = 0.25 \times \$10\text{M} - \$1\text{M} = \$1.5\text{M} \]
That’s $1.5M expected profit on a $1M investment (150% ROI). Every firm would salivate over this, but then, precisely no one outside of some venture capital firm is likely to evaluate this project on this basis. For the manager tasked with shepherding this nascent project to its desired destination, the calculus is quite different. Let’s say, in the case of a successful outcome, there’s a coin-flip’s chance of a 20% raise12. Alas, in the case of a failure, let’s assume there’s a 50% chance of losing one’s job and incurring six months of lost salary in the attempt to locate a replacement13. This is a wholly different equation:
\[ \operatorname{E}[\text{manager's outcome}] = 0.25 \times 0.5 \times (0.2S) - 0.75 \times 0.5 \times (-0.5S) = -0.1625S \]
Here, S is the manager’s annual salary.
This poor manager cannot access the overwhelming majority of the upside in success, but might pay dearly for failure. For far too many senior professionals, it is this calculus that demands either incrementalism, narrow initiatives with high probabilities of success14, or the political positioning wherein a project can fail without its leader being tarred with the blame.
And we wonder why nimbler challengers are capable of disruption while larger incumbents cannot? The former aspires to the desired solutions of the first equation while the latter aims to avoid the consequences of the second.
Do you want your best people to pursue high-upside projects? Ask yourself: “what equation are they solving?”
Whataboutism
There are games we should play and some games that we shouldn’t
Is “ought not” a thought caught that which we couldn’t?
Are hamp’rings and hind’rings of meaningful action
Just whatabout read-outs of obstinate fractions?
Before hands are raised in dismissive objection,
Is there hope that remains in a partial selection?
My primary livelihood has been some amalgam of data science, analytics, and FP&A, but my data science origin story begins with an algorithm to forecast the outcome of baseball games. Back in the early 2000s, historical data arrived on a CD-ROM in a padded envelope delivered to a campus mailbox cluster. Other data were procured via scraping performed by a dear friend who would later complete a doctorate in quantum computation and spend a career solving the type of quantitative problems that have yielded two decades of rich conversations across three continents. And a third compatriot, Jersey-bred, street-wise, and as kind as he is sharp, offered a real education to the bookworms in the dorm beside him. Twenty years later, he remains a blessing as both a friend and a sanity check for any would-be deployment of capital.
Why do I bother to share this heartwarming tale of collegiate hijinks turned career aspiration?
Because the model solved many problems, but not every problem.
In the ancient days of box scores and encyclopedias, a pitcher returning from injury was an unknown quantity. Would his fastball show life? Would his elbow endure another couple dozen curveballs? This was unknowable to even the most well-calibrated algorithm. We conceived of a mathematically sophisticated solution.
We just didn’t bet those games.
However obvious that revelation may seem, all intellectual structures are characterized by circumstances to which they apply, and circumstances to which they do not. A baseball model offers no insight during football season.
A few years later, in an office in the World Financial Center, two college classmates and I joined forces to co-found a hedge fund. Our first day of live trading was September 2, 200815. The days that followed were chaotic and horrifying: three well-educated twenty-somethings faced a harsher reality than we had imagined. Much to my satisfaction, our models endured the calamity16. How was this done? What mathematical wizardry ensured their survival? I consider this solution among the greatest intellectual accomplishments of my life… We named this particular bit of code the “Oh-shit-o-meter.”
It had one job: check the macroeconomic conditions, compare them with the historical conditions under which the trading models were calibrated, and, if they differed sufficiently, put the money in cash and sit on one’s hands. Initially, crusty, battle-tested traders came by our office to deride our lack of intestinal fortitude17. As October came and went, the markets crashed, and some of those same weathered faces were asking different questions about our discretion.
A model needn’t be insightful in all seasons to add value. A project needn’t solve every conceivable problem to be worth undertaking.
How many meetings end with explanations of why a solution will be inadequate or enumerations of its shortcomings? Can this solution add value in some cases? Is the cost of generating that value lower than the expected gains? Are you really planning to leave upside on the table because it cannot model injured tendons or subprime collapses?
Footnotes
Infamously?↩︎
The tennis lexicon for the scenario in which the server can win the set by winning the current game (in which they are the server).↩︎
https://www.tennisabstract.com/cgi-bin/player.cgi?p=AlexanderZverev↩︎
#ScrollingMyLifeAway↩︎
Not that I’m wholly misanthrophic, but I do often wonder how many of the seven deadly sins one commits within a single algorithm-driven rabbit-hole.↩︎
The apt term is a “fat-finger error”↩︎
Though having a human in the loop minding the model ain’t a bad idea.↩︎
I mean, we can still eat the snacks and wash them down with a can of Monster in the passenger seat!↩︎
You didn’t think those air-conditioned monstrosities sustain themselves in the desert by booking bets the customers typically WIN, did you?↩︎
Yes, I called you “Shirley.”↩︎
The alternative was a more interesting lead-in, followed by decidedly more complex mathematics…I knew you’d see it my way.↩︎
Most employees would covet such probabilities of advancement in title or salary.↩︎
We’ll assume that the next job matches the salary of the current job, despite the somewhat rosy nature of that supposition.↩︎
Or an elegant chart in a powerpoint deck implying that some metric moved in a positive direction to present the veneer of success (yes, I’m cynical, deal with it).↩︎
For those who remember their financial history, the bankruptcy of Lehman Brothers became official on September 15, 2008, with the purchase of Merrill Lynch (September 14, 2008) and the bailout of AIG (September 16, 2008) flanking that fateful date.↩︎
That said, I describe these days as akin to playing poker aboard the Titanic. One can certainly play hands well and, in so doing, amass chips…but the boat separating one’s arse from the unforgiving sea below is still the last place one wishes to be.↩︎
Our disposition was described using a word for both felines and female anatomy.↩︎