The Role of Intuition in Data-Driven Decision Making
Western philosophers, from Plato to Descartes, believed that decision making was a rational thought process. Our emotions, our hearts, and our intuition have no place in sage decision making.
Scott Adams, a popular modern philosopher, seems to be a fanboy of Descartes — when Dilbert calls his boss's penchant to ignore data for intuition, a slippery slope to witchcraft.
The idea that intuition is a slippery slope to witchcraft is carved in the hearts of all who have spent time making product decisions in a matrixed environment of a technology company.
Yet, in his popular and inspiring commencement speech, Steve Job proclaimed “Have the courage to follow your heart and your intuition.” Was he going against traditional wisdom?
Recent studies show how intertwined emotion and reason are. When emotion fails, reason fails. We also know that data-driven decisions lead us to local maximum, and without the help of intuition or product sense, taking the product to the global maximum is a losing battle.
Data-driven decision making and experimentation help with incremental product improvement, rather than building breakthrough products. To build a new product category, we need to rely on intuition and vision.
Savvy professionals know that without intuition, decision making becomes very sluggish, and, at worst, comes to a halt, in a complex environment with a thousand moving parts.
What is going on here? Why is intuition at once our friend and enemy?
Are Scott and Steve at odds with each other? Or, is there a unifying thought beneath this seemingly contrasting take on the role of intuition in decision making?
The field of psychology has some clues. One framework looks at intuition as ‘non-sequential information processing,’ while another framework defines intuition as implicit knowledge that is relayed to the conscious mind, from the unconscious mind. Kahneman and Tversky, authors of Thinking Fast and Slow, believe that intuitive judgments are derived from “informal and unstructured mode of reasoning.” With more research, we’ll have more ways to look at intuition, and we will have a better grasp of it too.
However, the underlying learning from all the research is the same, resonating with the assertion of Hippocrates, the father of medicine: We are what we eat.
If our body is nothing but what we eat, then our thoughts are nothing but what we consume.
Moreover, the whole is more than the sum of parts. Our mind does not just do 1-to-1 mapping of experience and decision; it creates a multi-dimensional universe out of all experiences, and intuitive decision results from multitudes of nodes from our experiential universe.
Our Intuition is ‘Khicadi’ (Stew) of our past experiences; unfortunately, we don't and can’t, with contemporary technology, know the recipe of the Khichadi.
Intuition is merely a data point
Not for the love of science, but for selfish corporate interest, I prefer to call intuition a data point.
Intuition risks becoming witchcraft when we treat it like an opaque black box. When we take the ‘Khichadi’ as it is, and don’t take a moment to think of the ingredients and the recipe, intuition often becomes a recipe of disaster. Leaders and decision-makers use it as a crutch to avoid difficult thinking. Under the self-afflicted weight of a lifetime of experience, leaders feel pressured to make intuitive remarks when they have none. Remember, not everyone can draw an Atwood-duck.
On the other hand, learning to look at intuition as data, not a black box, fully aware in certain cases it is a brown box, makes it open for interpretation. The new paradigm, intuition-as-data, leads to self-reflection, first-order thinking, and engaging debate — a recipe for robust decision making.
Thanks, Ricky Q for proofreading and editing.
