False Start: Why We Start Projects Too Soon
You can’t implement the solution (but you CAN build a model!)
Story #1 - “Sittin’ on the dock of the bay, wasting time…”
When Otis Redding recorded those lyrics in 1967, he did not know that his life would end a few days later, nor that his first single to reach number one would become a posthumous honor. Would he have sung so cavalierly about “wasting time?”
My first data science experience arrived a few months after following the completion of my undergraduate education. As a graduate student in environmental engineering who hadn’t taken a biology course since 9th grade and hadn’t seen the inside of a lab since freshman year, it was rather obvious that my advisor had admitted an atypical student. I emerged from a university department that, in 2006, graduated roughly fifty students, and sent them all to prestigious positions in investment banks or “Big Three” consulting firms. All except one, that is. I packed my bags for Champaign, Illinois to calibrate mathematical models like the rest of my classmates. The difference (apart from the compensation of course) was that mine were not intended for financial asset valuation or predicting customer retention.
The project had an impressive sounding dependent variable, but an incredibly simple objective. Funding had been secured to study “Benthic hypoxia” (low oxygen levels near the floor of a body of water) in Corpus Christi Bay, TX. The procedure is easy enough: Load up a boat with a few sensors, a professor, and a couple day laborers (ahem, “graduate students”), take some measurements, return to shore, analyze, publish manuscripts, etc.
The rub was that hypoxia is a transient phenomenon, and it’s rather dispiriting to awaken before sunrise, spend time and energy schlepping the boat, the equipment, and the human beings, only to take measurements where and when the oxygen levels are within typical ranges. Enter the graduate student with the modeling chops and the uncallused hands! Write a model that predicts, in space and time, where and when hypoxia is probable, then, if the model suggests there will be something worth measuring, tell everyone to set their alarm clocks and hit the docks the following morning.
Before I arrived on the scene that fall, a more senior graduate student had taken a crack at this problem. She was decidedly more sophisticated with respect to her modeling acumen, devised a complex set of differential equations that tracked the motion of sediment, oxygen, salinity, heat, and heck knows what else. Her knowledge of aquatic physics was intimidating to a guy still frightened of embarrassing himself among the academic elite he aspired to join. The mechanics she had expressed in code were complex, descriptive, and well-researched over months and years.
But this was 2006. Amazon was a place to buy books, not to spin up extra computational resources. And on the Flintstone-esque laptops of the age, her model required roughly 48 hours to execute a single prediction. So when, at 3pm, a professor would ask for an estimate of the probability of hypoxic conditions across the bay the following morning…the model would deliver its answer more than a day after its information might inform a decision.
This might have been fodder for scholarly publication, especially if its predictions were more accurate than predecessors, but it wouldn’t solve the problem at hand. If the goal was to avoid waking up, lugging sensors, and loading boats in vain, a different approach would assuredly be required. Within a few weeks, I had rigged up a decidedly simpler machine learning tool, the kind which would probably could be implemented before the second cup of coffee by a data scientist circa 2025. It broke precisely zero new ground quantitatively, nor did it deploy cutting-edge understanding of the hydrologic chemistry in play.
In plain English, the model simply searched a database for historical examples where salinity, wind, and temperature conditions were similar to the moment in question, made a few adjustments for seasonal and daily patterns, checked where and when within those examples, there was hypoxia the following. It performed some simple probability computations and even on a laptop more accustomed to ethernet cable insertions than modern predictive modeling, the runtime was roughly the quantity of time required to refill one’s beverage and procure a snack.
The ability to understand a domain well enough to describe the governing dynamics and the ability to write software to make a prediction is not necessarily the ability to solve the problem. It is unwise to invest simply because one can build a model. Does that model actually solve the problem? Do you understand what must be true of that solution for that solution to have any tangible utility? Or will you write a model that makes spectacularly impressive predictions no one can actually use?
The aspiration is wise investments, not accurate models. Otherwise, we’re just sitting, wasting time.
Story #2 - “And you’re still the same…”
When Bob Seger penned those words about his old friend, the wise gambler, he recognized that with each encounter, each iteration, the friend was “still the same.” Our personas update over the years. Our addresses, phone numbers, jobs, even names can change. And yet, in the end, we are still the same individual, even if a database full of our descriptors varies with the passing of the years.
Why am I waxing poetic about the nature of change and fundamental continuity of self? Isn’t that discussion better reserved for Derek Parfit and other philosophers? Perhaps. But if a corporation retains a database full of human beings, and those human beings proceed through the undulations of their professional and personal life, it is generally preferable to keep their records up to date. Of course, corporate data hygiene is rarely without issue, and from time to time, a new entry is created when the person already exists in said database, albeit with slightly different characteristics from a prior moment in time.
Soon, the number of individuals stored is measured in millions and it is certain that some of those entries are actually duplications of earlier entries rather than different human beings. Thus begins a classification exercise wherein potential pairs are explored. After all, it is extremely difficult to request anything from an individual with outdated contact information and when two profiles represent the same living creature, we’re bound to err. One simply explores possible duplications, devises an algorithm to assess whether the two profiles are or are not the same person, performs some laborious cross-referencing to validate performance, then we deploy the solution, right? Of course not.
As those familiar with statistical error, one more consider the consequences of false negatives and false positives. In this case, a false negative would be the case wherein two profiles really do reflect the same individual, but the algorithm fails to detect it. This might be the case where a person has changed their last name in marriage, moved across the country, entered a new field, changed their phone number and email, etc. Sure, it’s actually the same person, but any algorithm that doesn’t ask each entry to coffee to recount their last couple decades would fail. The database is exactly as messy as it was previously and life continues. A false positive, on the other hand, when we merge two profiles for two wholly distinct people. In one particularly entertaining case, there were two NYC-based cardiologists who shared the same first and last names…and since they were employed by the same, massive hospital system, they shared the same work phone number and office address. The only means of revealing the improbable result was a manual inspection of photos on public-facing hospital websites.
False positives in this case are costly. Merging two profiles for two different people means, at best, partially incorrect information about one of the two individuals and at worst, some non-trivial liability if we present that individual to another prospective client and in so doing, assert attributes he or she does not possess.
After months of research, the team had developed a tool with a false positive rate in the ballpark of 1 in 100,000. (The false negative rate was less extreme, but as discussed, some of those errors are inevitable and of minimal cost). When the solution was refined, polished, and ready for primetime, the project’s manager prepared for integration with the company’s existing data engineering operations. This did not go well.
Firstly, the legal department was unwilling to accept a false positive rate that, though mathematically impressive, would inevitably yield tens or possibly hundreds of such cases. Secondly, the merge algorithm, however meticulously considered, would violate some previous immutable rules upon which business intelligence ultimately relied, most notably, the timing of certain actions always preceding certain other activities. Merging two profiles created years apart could absolutely result in a user journey wherein event B happened before event A because event B never actually occurred for the newer profile, but did occur for the older, invalid profile, and merging a profile without noting that it had performed action B at least once was also problematic…and this continued in multiple meetings until several data-minded individuals wanted little more than to defenestrate the combination of lawyers and managers they held responsible.
Sure, we could build a model. Moreover, we could build a model with statistical properties that would impress engineering managers. But the project involved liability risks and costs the legal department was simply unwilling to bear. The analytical implications of the new and “improved” dataset were likely to generate confusion for which the C-suite had little appetite. It solved a problem. It created others.
Unless the solution’s downstream costs (and I promise, that value will be greater than zero) are markedly smaller than the solution and the relevant stakeholders are willing to absorb them, the ability to build a suitable model is little more than academic aptitude. The engineers’ salaries were paid and that investment yielded little more than a few charts demonstrating the impressive error rates.
And for the frustrated engineers who calibrated and validated a model that will never see the light of day, the data are still the same.
Story #3 - “Don’t go changing to try and please me…”
When Billy Joel sang these lyrics to a fictionalized woman, he asserted that he did not want “clever conversation,” that he would prefer not to “work that hard.” In fact, he tells her, “I love you just the way you are.”
Any modeling exercise is fundamentally an investment in an improvement of the status quo. This, of course, requires the type of situational awareness most organizations lack. During my second chapter in Champaign, a colleague of mine was assigned a most unfortunate topic for her doctoral dissertation. The problem statement was simple enough.
Beneath the streets of Gotham (fine, Chicago) lie a series of sluice gates that can be raised or lowered to divert storm water and sewage into temporary reservoirs for temporary storage. This reduces risks of urban flooding and minimizes the likelihood that raw sewage will find its way into Lake Michigan. My intrepid, fellow PhD candidate was tasked with calibrating a predictive model that, given some forecasts of precipitation and the current quantities of water in the system, would choose when to open and close these gates. Flooding would be averted, Gotham would be saved, she would depart a conquering hero (or at least having successfully defended her doctorate).
The rub? That system of tunnels and tasks was hardly new. The decisions to open and close gates at that time (circa 2010) were the purview of the Chicago sanity district. In other words, two salty old engineers, who have witnessed every horrifying sequence of midwestern hydrologic misery and frozen climates of the prior decades. Since Chicago has not descended into the noxious, polluted anarchy of an Upton Sinclair novel nor the flooding-driven humanitarian crisis of post-Katrina New Orleans, we can conclude that these engineers have been competent. As the current solution takes the form of a couple municipal employees, we can safely conclude the cost is probably far more manageable than an army of consultants.
Thus, when the doctoral research begins, the real question (not the one my colleague repeated at conferences) was “can this be done more effectively and affordably than these two crusty, savvy engineers in their cubicles?”
Moreover, most of the decisions are not that complicated for savvy human beings watching the system on a daily basis. They know which tanks have capacity and which do not. They know the weather forecasts. They know how to navigate scenarios that in all likelihood, they have seen previously. And if, perchance, something wholly unique occurs, the proverbial “black swan” is by definition, beyond the scope of the data on which such a model was calibrated.
The combined sewer overflow algorithm my long-suffering, all-but-dissertation colleague attempted to improve upon semester after semester offered nothing of value (publication or otherwise) unless it improved upon the existing solution. The existing solution was probably sub-optimal, but entirely sufficient. Whatever frailties the Windy City harbors in their politics and their budget, they manage midwestern thunderstorms with reasonable aplomb.
In this case, the underperforming investment was that of research funding expected to yield scholarly publications and prestige, but this form of pre-ordained failure is now commonplace in the race to AI-enable every imaginable process. If the existing process is affordable and efficient, the bar for wizardry-driven becomes impossibly high. What process do you intend to change? And who exactly are you trying to please?
You’re swinging at the wrong pitch (of metrics and madmen)
My father, within the canon of quips known to retired math teachers and other nerd-adjacent dad jokes, often referred to “TLAs.” What is a TLA, you ask? A TLA is a “Three-Letter-Acronym.” The joke being, for those who haven’t already nodded off, that “TLA” is, in fact, a TLA. This yawn-inducing digression seems silly, until we realize that the tyranny of TLAs is now we will determine if the investment of time and treasure has paid its desired dividends.
Each project will have its OKRs, KPIs, NPS, CAC, LTV, ARR/MRR, or other terms beloved by MBAs. (If you were easily able to expand the initializations above, my condolences on a professional life that has contained more than its share of powerpoint decks. Otherwise, I commend you on a life in which you’ve probably helped deliver tangible goods and services to real human beings!) The worthiness of any hypothetical investment will assuredly be tied to its likelihood of raising or lowering some metric on some dashboard managed by a team of unsuspecting analysts who have no skin in the game. Far too frequently, the metric moves…and nothing of importance actually changes.
Before delving into some dull, corporate example, let’s consider our national pastime and its most obvious metric - balls and strikes. Clearly, the objective is more of the latter and fewer of the former, right? Balls lead to walks, strikes lead to strikeouts. Develop a model to minimize balls and maximize strikes, ship the code, and wait patiently for your World Series ring!
So why don’t major league pitchers throw the overwhelming majority of their pitches over the plate? Even for those who are not avid baseball fans, the logic is clear. Throw every pitch over the plate and batters will no longer be required to make that split-second decision whether or not to swing. Thus, the proportion of pitches in the strike zone is one important metric, but not the only important metric. A single-minded focus thereupon might yield an aesthetically-pleasing dashboard, wherein the “IZP” (In-Zone-Percentage) heads steadily up-and-to-the-right, but the opposition will score more runs.
Modern baseball has discovered that throwing 100 mph fastballs and sweeping breaking balls with an unpredictable sequence of accessible and unreachable locations is an exceptionally effective strategy for avoiding solid contact. If the number of walks increases a bit and pitch counts rise, diminishing the number of balls put in play will be more than worth the cost, much to the chagrin of fans and MLB marketing executives.
This is a contrived example, and yet the principle of Goodhart’s law applies more often than we might realize. This law (to the extent that British economists create “laws”) states that “when a measure becomes a target, it ceases to be a good measure.” Minimizing bugs in software is a worthy objective. Turn minimization of bugs into the key metric upon which engineering teams will be judged and every line of code will be subjected to an infinitude of interminable reviews. Diminishing the number of fraudulent transactions is a worthy objective. Offer a bonus associated with this indicator and someone, somewhere will have an incentive to enter a world where no one transacts.
The optimal quantity of bugs is not zero (otherwise, we could never ship code). The optimal quantity of expected cybersecurity breaches over the next century is not zero (why not simply lock the doors, shudder the business, and prevent any employees from accessing their terminals?).
Before any significant investment of dollars or hours occurs, we must first understand:
How a solution might be implemented
The downstream impact of that solution
Whether that solution would actually add any value beyond the status quo
Whether the metric that solution would improve is actually valuable (as opposed to easily gamed/manipulated)
How many thousands of hours would be spared by considering the answers to those four questions and throwing a yellow flag upon the table for a “false start?”
Strikes good, balls bad…until modern baseball led to 100mph pitchers and hitters swinging for the fences
Minimize bugs or security breaches (just don’t write or ship any new code…and lock everyone out of the office)