Heuristics – Why life is too complex to account for everything
I first learned the word ‘heuristic’ from Daniel Kahnemann’s phenomenal book “Thinking, Fast and Slow.” In that book, Kahnemann identifies two different mental modes we operate in – System 1, and System 2. System 1 is a more ancient system in our brain, refined by the crucible of evolution to make snap judgments very quickly – the goal is to get it mostly right, most of the time, and it does an excellent job of this. System 1 operates largely automatically, it often had to, to keep us alive in the dangerous world we evolved in. System 2 is a more recent addition to our brain, one that contributes to our higher order reasoning – our capacity to stop, slow down, and think about things rationally. It provided us the capacity to think ahead, make complex plans, and do cool things like mathematical reasoning.
The important thing to note here is that System 1 is a very heuristic-based system. It’s designed to take an immensely complex world and simplify it such that in a split second, it can make a decision – many of these decisions were (and still are) life or death, and it makes complete sense why, even if they are flawed, they have come to us today. Kahnemann’s book explores in detail the many heuristics, which he calls cognitive biases, that we have inherited.
We have been conditioned by our modern environment to hear the word bias and draw a number of associations with unfairness, prejudice, injustice. Let’s be clear – a bias is indeed a prejudice. Almost by its very nature, our System 1 heuristics act faster than – and prior to – our slower System 2 system responsible for rendering a reasonable judgment. It is pre-judging. And the reality is that that these snap judgments, these biases, are mostly right, most of the time, for the environment they evolved in. Kahnemann’s book explores in details the myriad ways these judgments can go wrong, especially in environments far removed from our evolutionary past, but I just want us to hang onto an important takeaway here: Heuristic thinking has survived in evolutionary terms so well because it has been instrumental to us in navigating a world with so much complexity. We have never been – and still are not – equipped with the computing power to reason through complex reality with any degree of efficacy, and most definitely not with any degree of speed. We need simple rules that get it mostly right, most of the time, and we should leave it to our capacity for reason to simply identify when these heuristics are failing us and rather than replace them with logical system-2 thinking – we ought instead replace them with better heuristics.
Chaos theory is the study of complex systems that become rapidly unpredictable due to the vast number of interacting parts. If you’ve ever heard of the butterfly effect – often cited in time-travel fiction wherein even the minutest change in initial conditions can spiral out into vastly different outcomes – this is Chaos Theory at work. It demonstrates that our efforts to try to model complex systems – like weather systems and global economies – are virtually futile over a long enough term, as even minute changes can radically alter an outcome. Nassim Nicholas Taleb identifies certain large-scale unpredictable events as Black Swan Events, and he notes that while these events often seem predictable and ‘solvable’ with the benefit of hindsight, they are virtually impossible to account for. We create post-hoc narratives to explain Black Swans, but they are rarely useful in addressing future events. The Coronavirus Pandemic is a perfectly good example – the event of a global pandemic was perfectly predictable, and predicted, but the actual effects of it were impossible to appropriately plan for. There would have been no method short of a perfect simulation of reality that could effectively predict where the pandemic arose, under what conditions, what effect the political environment had on the response. How could anyone have modelled in the early 2000s what potential effect a Donald Trump presidency or the Joe Rogan podcast might have had on the outcome of a global pandemic? If you were such a prophet, who on earth would have believed you? What on earth is a podcast, anyway? And yet in the early 2000s we were modelling and predicting plenty of things about the year 2020, with plenty of confidence.
Trying to use a rational long term strategy to predict the future is, to use Taleb’s colorful description, a “sucker game”. It’s just not effective, because too many things can change. The more detailed your prediction, the more complicated your plan, the more likely one single random event could shatter it. So instead, I propose, we are better served with heuristics, rules of thumb, simpler models that we acknowledge are imperfect. Again, do not take a heuristic and treat it as a law, as we are so prone to do – we so often take the world and distort our view of it so that it can fit our clean models. To borrow another wonderful comparison from Taleb, this is a “Bed of Procrustes” problem. Procrustes was a figure of Greek myth, who invited passers-by to stay with him and sleep in his bed – however, he either stretched them or amputated them so that they fit perfectly. This, we might acknowledge, is a failure of hospitality. Let’s take similar care with our models not to decapitate reality to fit the bed, and pretend as though we are good hosts.
So rather than spending endless hours theorizing, let’s just consider some interesting and useful heuristics, quotes, and aphorisms that we could apply in our daily lives. I’ve found that holding these simple thoughts near to hand make life much more manageable than the constant (and futile) desire to rationally account for all outcomes step-by-step. Some will contradict one another, or be inapplicable in certain contexts, and that is okay – remember that these are not natural laws of physics – even the mathy ones – but rules of thumb. And just because a rule is called a rule, doesn’t mean it’s a rule, you get me?
Quotes, tools, and “rules” to live by:
“A failure to plan is a plan to fail.” – Benjamin Franklin (apocryphal).
I couldn’t resist a contradiction out the gate – wasn’t I just critiquing the efficacy of long term strategic thinking? Indeed! And let me provide another aphorism I enjoy:
“No plan survives first contact with the enemy” – Helmuth von Moltke. Or perhaps you prefer “Everyone has a plan until they get punched in the mouth.” – Mike Tyson
It turns out that despite the general futility of planning, some degree of planning is still indispensable. I hold both of these contrary statements in my head as a reminder to myself that it is worth going into any endeavour with a plan, but that you must not spend too much time planning. Let me explain with another invaluable heuristic – the principle of diminishing returns – “The benefits gained from something will become proportionally smaller as more resources are invested into it.”
Namely, planning is instrumental, up to a point. Eventually – often quite quickly – you reach a point where you have covered the most likely outcomes, and any further attempts to address complexity are proportionally more futile, because the world is intrinsically, deeply unpredictable. The branching pattern of cause and effect will create exponentially more things for you to plan for, the deeper you get into planning. If you’re going into the ring with Mike Tyson, planning a little makes sense – you want to train, you want to be aware that his knockout punch could blow your head off, and you want to understand his cues to whatever extent possible. But you are not going to be able to predict every punch he will swing to the second he plans to swing it, and train by choreographing your perfect dodges. We know that’s absurd, yet we try to do it with far more complex systems all the time.
Let’s distill this idea into an excellent quote from Dwight D. Eisenhower, who was in charge of the Allied High Command during the Second World War:
“Plans are useless, but planning is indispensable.”
Now let’s explore another heuristic, the Pareto Principle (also known as the 80/20 rule):
80% of the outcomes come from 20% of the causes. This ‘rule’ manifests itself with a certain startling mathematical reliability in many contexts. For example, having developed several applications now, I consistently notice that a small number of users use our app dramatically more than the others, often following this same curve. This principle has also applied to income streams, where 20% of the customers provide 80% of the revenue, or 20% of the ads provide 80% of the conversions. 80% of your complaints come from 20% of the customers. You get me…
This is a really useful thing to keep in your back pocket. It demonstrates two (once again seemingly contradictory!) ideas:
Idea 1: Distributing your attention equally across all options is not, as a rule, a good use of your time, because some options are going to provide a much greater return than others.
But ALSO:
Idea 2: Distributing your attention equally across all causes early on in a process can in fact be an excellent use of your time, if it enhances your potential to discover which options provide outsized results.
This also relates to bad outcomes. This was an insight by Tim Ferris – suppose 80% of your complaints come from 20% of your customers. Also suppose that 80% of your income comes from 20% of your customers. To any degree that those complaining nuisances are not your top customers (they usually are not), they might well be a complete waste of your time and effort. Fire them. You don’t need their business.
We can also distill this into another heuristics: Explore widely early on in a process in order to generate a good list of options, so that you increase your likelihood of selecting an optimal option. In the book “Algorithms to Live By” this is encapsulated in the “37% rule” – wherein you spend 37% of your time exploring options without committing to any. After that point you should be prepared to commit to the first option that surpasses all the other ones you’ve seen before. The key insight here is that as you progress through the list of options, all else equal, your chances of finding the optimal one actually decrease the further down the list you go as you approach your deadline. Your second option may have a 50/50 chance of being better than the first, while the 10th option has a 1 in 10 chance of being better than the preceding 9. There is an optimal stopping point wherein you are better served making a decision than wasting too much time and energy (especially if either are limited, which ultimately they always are, you poor mortal) trying to select the ‘perfect’ option. Does this mean you will sometimes miss a better option? Yes. But there is a cost – a very pricey one – associated with perfectionism… Which brings us to another wonderful quote.
“Perfect is the enemy of the good.” – Voltaire. Oh I love this one. All of us have encountered perfectionism – some of us are guilty of it, others just see it in others and groan. Perfectionists have a terrible habit of never getting anything done, here is another I like:
“Perfectionism is procrastination.”
There is a difference between holding yourself to high standards and failing to deliver because you’re ‘not ready’. Perfectionists often inhabit the latter category, they put off finishing anything because they feel like they may be missing better options, or they fear judgment of their flaws. The problem is that usually cost of perfection is incredibly high – consider again the principle of diminishing returns and the Pareto principle. There is an inversion of this Pareto principle I often like to think of when it comes to project management and planning:
“80% of the time is wasted in the last 20% of a project.” – Or perhaps, “The amount of time taken is inversely proportional to the amount of work that is supposedly left to do.” Whenever anyone says something is “99% done,” I often suspect it’s going to take months to complete. There are always countless little things, small corrections, new ideas. “Nearly done” is purgatory for project completion. Beware this hellish place. Another ‘law’ in this domain is:
Parkinson’s Law: “Work expands to fill the time available for its completion.”
We’ve all experienced this. Nebulous far off deadlines where nothing gets done, or at least nothing useful (lots of ‘planning’!) only for everyone to get their shit together as soon as time is tight, and getting most of the work done in a fraction of the time. Usually by throwing out the plan.
Hard deadlines, especially if there are consequences (either incentives or punishments) get results. I suspect it’s because tighter deadlines enforce some of the heuristics we’ve been discussing. With a tight deadline you do not have the luxury of taking a month to carefully design a plan – you make a basic one, which is usually better anyway, because 80% of the time spent on a month-long planning process was probably spent on frivolous or unpredictable details and futile, paralyzing risk analysis. With a tight deadline you must rapidly select options and get to work – and if something isn’t working, you need to change tack fast. You don’t have time to be a perfectionist, so you focus on what’s effective, not what’s ideal. We’re actually remarkably effective at getting to the meat of the problem, because, as they say, “Necessity is the mother of invention.”
Here’s another pair of quotes:
“50% of all quotes online are misattributed to Benjamin Franklin” – Benjamin Franklin
“50% of all statistics are made up on the spot.” – Benjamin Franklin
Forgive my frivolity. These two quips remind me, first of all, not to believe everything I hear, especially in the domain of pithy quotes and statistics. Don’t just believe me when I quote people, I’ve misquoted people before and I’ll do it again, damn it! And as for statistics, statistics are useful, but they are used extensively to mislead – even the ‘true’ ones can be deeply deceptive.
Mark Twain once said (allegedly…), “There are three kinds of lies: Lies, Damned Lies, and Statistics.” Oh, he attributed it to Benjamin Disraeli, apparently – who may never have said it. Ha ha. Where were we going with all this? Oh, right, don’t be so damn trusting. If it matters, check your sources, take a minute and verify. While it’s impossible to go and verify every thing you hear, and sometimes you really do need to take someone’s word for it, it pays to be constantly aware that it’s a lot easier for a statement to be untrue than it is for it to be true. In fact there are virtually countless ways for things to be false and a very finite set of ways for them to be true: Making up bullshit is far easier than finding the truth, and often quite profitable. So beware.
“Simplicity is the ultimate sophistication” – Leonardo Da Vinci
And yet another – “Brevity is the soul of wit.” – William Shakespeare
This bears constant reminding. One of my hobbies the last few years has been constantly meddling with productivity systems, often to my own detriment. I’ll come up with all kinds of convoluted ways to track my progress and manage my notes. Databases, schedules, automations, all of it! It’s messy, and it breaks easily. And oftentimes I’ll spend so much time tinkering and fixing that I barely get anything real done in the first place – it’s not work, it’s work about work. Nassim Nicholas Taleb would call these systems of mine “fragile”. The more complicated a system becomes, the harder it is to maintain, and the more prone it is to breaking when exposed to any sort of volatility. It begins to require more energy and resources just to keep it functional, you need to remember how it all works, and if something goes wrong, you need to diagnose it. I became something of an expert about my overly complex Notion database system. I could have written a book on it, and sounded very smart. We’re prone to mistaking something complicated for something sophisticated, we think that because it takes an expert to understand it, it must be very refined. Not so! These things often just obscure the weaknesses and failures of a system – when a layman can’t understand it, it makes it easier to simply tell them to stand aside and let the experts handle it. Many faux-experts and intellectuals drape their knowledge in mountains of jargon and big words to sound sophisticated.
Well, as an expert on my convoluted Notion system that probably would have been more use to me in the form of a notepad, let me tell you, the experts aren’t always all that clever. The best scholars are those who can (accurately) describe something complicated in simple terms. But don’t take it from me: “If you can’t explain it simply, you don’t understand it well enough.” – Albert “smarter than you” Einstein
Now here’s a tried and true design principle for you – “Keep it simple, stupid!” Here’s another by Lao Tsu: “To attain knowledge, add things every day. To attain wisdom, remove things every day.”
Moving on… On the topic of removing things every day, let’s just remember:
“Don’t throw the baby out with the bathwater.” – I say this a lot, it basically means take care not to get rid of something valuable along with something unwanted. Another close cousin of this idea is known as “Chesterson’s Fence” – “Do not destroy what you don’t understand.” It’s called ‘Chesterson’s Fence’ because of a parable, wherein a man comes across a fence in the middle of a field. The fence is in his way, and so, because his goal is to cross the field, he smashes the fence and proceeds along towards his destination. Instead he should have considered – why is there a fence in the middle of this field? If you can’t explain why, do not be in such a hurry to knock it down.
Say he went to ask the old man who put it up, who tells him that he built it 60 years ago to keep wolves out. If wolves have been eradicated, nowadays this fence may just be an obstacle – trusting of course there aren’t other predators to be kept at bay. But if the fence is to keep his sheep in, and he still has sheep, then knocking down the fence would be a bit of a dick move.
In many old and/or complex systems, the answer can’t be provided by the fence-builder – he’s either dead, or just as likely, it was built collectively by countless people none of whom had the full picture. So proceed with caution when knocking down rules, traditions and institutions that have been around for a very long time, because their purposes can be myriad, and not always well-understood. This impulse to knock down fences brings me to another idea…
“Short cuts make long delays.” – Peregrin Took (J.R.R Tolkien) – Sometimes it’s important to have patience, and not cut corners. In this talk of heuristics, quick rules of thumb to make rapid decisions, this may seem out place, but of course there are many circumstances where the quickest response is not the best! The quickest and easiest short-term solution often creates very time-consuming downstream effects. An inversion of this comes in a little nugget of ancient wisdom, a favorite of the Roman Emperor Augustus Caesar, and a motto of the Medicis:
“Festina Lente” – “Make Haste Slowly”
This speaks to the importance of addressing tasks with the appropriate combination of urgency and care. It’s sometimes translated as “More haste, less speed”: don’t be rash! This saying inspired the fable of the Hare and the Tortoise, wherein the Hare rushes, but the tortoise “with a prudent wisdom hastens slowly.”
Another quote that helps illustrate the importance of taking care is a beautiful remark by the astronaut Chris Hadfield: “There’s no problem so bad that you can’t make it worse.” – it bears reminding that rash and hasty responses to problems are often more dangerous than no response at all – Nassim Nicholas Taleb proposes that in many cases, procrastination and hesitation exists for a reason – to prevent us from doing something that may either waste our effort or make our situation worse. Just as I alluded earlier that there are countless ways for something to be false and a set number of ways for something to be true, this means: There are a lot of wrong answers to choose from, and only a few good ones. It often pays, except in matters of great and indisputable urgency, to stop, take a breath, and think for a moment, so long as you understand that some problems will be intractable and that more time spent contemplating will not solve them. And on the topic of making haste slowly…
“Never stay up late doing something you wouldn’t get up early for.”
I wish I followed this advice more often. It’s much easier to sacrifice sleep in the evening than in the morning, I find, and yet sacrificing sleep in general is a fool’s game. For one, doing something tired is almost always less effective – you would be better at whatever it is you wish to do in the morning after a good night’s sleep. And if you wouldn’t get up early to do it, it wasn’t worth doing.
Here’s another – “The best bridge between despair and hope is a good night’s sleep.” – Matthew Walker. Often when you are miserable, the best thing you can do is check if you are hungry or tired. If you are, addressing one of these things will almost always help. And downstream of this, when being aware of your emotional state: “Never make promises when you’re happy, never make decisions when you’re angry.” And if it isn’t actually your current state, but worries about your future state, that are making you miserable? Well…
“It is indeed foolish to be unhappy now because you may be unhappy at some future time.” – Seneca
And another in the same vein from the same man:
**”Two things we must therefore root out: fear of distress in the future and the memory of distress in the past. The one concerns me no longer. The other concerns me not yet.”
I often need to remind myself of this, when I worry about the future or the past. Again, the future is often so chaotic that it cannot truly be planned for, and worrying about every eventuality is not only a waste of time, it’s usually counterproductive. And the past has already happened – its only value is to help us plan for the future, and then only to whatever extent that is truly possible. It is better instead to invest in your own robustness, ensuring your capacity to face difficulties as they arise, rather than preparing in advance for what is intrinsically unpredictable. One insight we can gain from the past. “Make many mistakes, but never make the same mistake twice.“
Recall too that many things in life are outside of your control – to worry about them is a waste of time. This locus of control is fundamental to the Stoics. Epictetus, another Stoic philosopher, remarked: “It’s not what happens to you, but how you react to it that matters.” – or from William Shakespeare, “There is nothing good or bad, but thinking makes it so.”
Much of the misery that comes to us from life is from our perceptions of reality, not the reality. It is the stories we tell ourselves, feeding our emotions. Have you ever felt angry, forgotten about it, and found yourself reminding yourself that you’re supposed to be angry? Have you had fake arguments in your head to stoke your own annoyance? An excellent quote attributed to the Buddha, “Anger is like holding onto a hot coal with the intent of throwing it at someone else.” or another, “Anger has a honeyed tip and poison roots.”
And as for fear, fear of disaster, of a future we can’t control for? The anxiety that drives us to make as many plans as possible for as many outcomes, as fruitless as it often is? To Seneca: “Disaster is the opportunity for true worth.”
The Stoics have a philosophy that Taleb would describe as antifragile, that is, it benefits from volatility instead of being destroyed by it (fragility). And this quote from Seneca embodies this – to the Stoic, adversity was desirable, as it provided the perfect opportunity for one to strengthen their character. Instead of planning for every outcome, it is better to become the kind of person who can face any outcome.
And at the end of this list of what I hope were elucidating and useful quotes and rules of thumb, I have this one for you:
“Any fool can make a rule, and any fool will mind it.” – Henry David Thoreau
Remember, as always: Use your head. These are tools, not rules, no matter how many times they get referred to as laws or some such. Don’t try to twist reality into knots so they conform to your favorite pithy saying from the ancients – pick what is useful, and use it.
