Perhaps the worst-kept secret of the digital age is the fact that algorithms have increasingly come to govern our lives.
Each day, as we continue to use our devices for seemingly every purpose, we allow various third parties to access our data. This ‘Big Data’ movement, in turn, has become the competitive fuel for businesses doing serious market research.
Paint an accurate and comprehensive picture of your market, and you’ll reap the rewards. Whether you’re gathering your own data through an online system of survey collection or paying Facebook for audience insights, this has become the modern paradigm of digital marketing.
But in the end, all that information is only as good as the brains and discipline being applied to it. And many of those involved, from entrepreneurs and executives to marketers and analysts themselves, could use some training in a lesser-known form of reasoning called abduction.
Cause, effect, and relationships
Analyzing the data you’ve gathered will tend to bring up certain patterns. Relationships will always emerge, but we need to be careful before jumping to conclusions.
There exists a significant, yet often overlooked, distinction between causation and correlation. With the former, one event is clearly established to cause another. With the latter, there is simply a significant relationship, whether positive or negative, between the two.
Within some aspects of a data set, you can isolate and establish causation. For example, sales stemming from clicks on a paid ad platform or user behavior differences during A/B testing.
But as you try to process the big picture, more variables become involved, and it’s harder to state with confidence that one event truly causes the other.
For instance, retail giant Walmart partnered with The Weather Company and IBM to use weather forecasts to predict sales at their stores on a zip-code level.
Possibly, when a hurricane is forecast, people will stock up on Pop-Tarts that don’t require refrigeration and have a long shelf life. But causality is far murkier when the research reveals that shoppers buy more berries when the weather is cool and less windy. Here, you’d probably have to settle for a positive correlation instead.
A complexity problem
In effect, isolating causal relationships requires you to have definition and control. This is a well-known problem in experimental design. You need to rule out spurious alternative relationships between the two variables in question.
Imagine the demands of doing that if you’re a company the size of Walmart, with nearly 5,000 stores in the US alone. You’re attempting to analyze cross-sectional data of every product sold in the inventory of each store, tied to local weather patterns that vary daily.
It’s an immensely complex effort that may not even be able to yield causality, as in the case of the berries-versus-wind relationship.
You may not have to crunch data on such a scale, but you need to beware of the complexity problem. It can lead to a mistake along the lines of the green lumber fallacy. You don’t need to know all the details. You only need to have a grasp of what really drives results.
Reasoning differently
Some marketing professionals and entrepreneurs possess a certain intuition to this effect. To them, statistical analysis and data-driven research are useful tools, not replacements for critical thinking.
The difference lies in how they instinctively apply a form of reasoning most of us aren’t familiar with. We typically use either deduction, from general principles to specific conclusions, or induction, which moves in the reverse direction.
Abduction is what more people need to train in and apply in marketing and business in general. It involves working with existing data to quickly come up with the best possible explanation for those observations.
With abduction, less time and effort is invested in probing for causality. At the same time, you don’t just take blind leaps of faith and work off of questionable assumptions or spurious correlations.
Abduction involves placing a bet based on insight. The insight is more effective when you have domain-relevant expertise, making it an educated guess. Then you proceed to bet on your insight, taking action quickly in a manner akin to the startup’s mantra of ‘failing fast.’
Using abduction doesn’t devalue data or the expertise offered by analysts and marketers. It simply places them into context as sources of deductive and inductive reasoning.
In the end, focusing your efforts too much on one or the other leads to diminishing returns. What matters is that you make bets, and through trial and error, arrive at something that will give you market differentiation.
