Data analytics is now a $204 billion industry, according to Forbes. In India, it's expected to cross 16 billion mark by 2025 from the current $2 billion. Riding high on the boom sentiment, companies like Apple, Google are employing data analytics by the hordes. So much so that we have all started believing that scientific data analysis and predictive behavior are a must-have capabilities that lead to better business decisions, huge service improvements and product innovation.
In the words of Roger L. Martin and Tony Golsby-Smith, writing for Harvard Business Review, "When we face a context in which things cannot be other than they are, we can and should use the scientific method to understand immutable world faster and more thoroughly than any of our competition."
But is it really so?
Aren't there situations where reality is very different from its surface appearance? When a consumer says one things and implies quite the opposite? When initial fads fizz and fade out quickly from consumer consciousness. In such situations, predictive science, especially one dealing with human flirtations with brands, fails the litmus test of being an exact science. Data science may be telling one story, while the actual shopping behavior betrays another.
I once spoke to over a dozen-odd senior executives of large organizations.
The consensus that emerged is that behavioral analytics is emerging as a popular science but human behavior is very dynamic and non-predictable. Complete and total reliance on an analytical approach may not give us deep insights into human behavior, where intuitive intelligence must also come into play. Even in our social interactions, we depend a lot on intuitive intelligence in articulating and crystalizing our impressions of others, and accordingly, framing our thoughts. This is not possible will data-driven decision-making that's singularly devoid of intuitive intelligence.
Scientific methods are purposefully designed to understand natural phenomenon, where cause and effect are usually fixed elements of the equation. When cause changes, effect changes in the same proportion, and both are controllable variables, dependent on each other. But with human beings things are a little more complicated and baffling. A same cause can produce same effect, multiple effects, or no effect at all, as the case may be!
Time and other extraneous conditions also play a big role in who this effect gets manifested in human beings, in a manner that often defies scientific logic. Data is an outcome of "cause and effect" effected by external factors. Data is rooted in evidence and logic. In my opinion, this factor is the biggest impediment for big data analytics.
In my long career, I've personally experienced that most innovative and disruptive movements spring from the tendency not to accept the evidence which looks evident. Big milestone in business start with a gut feeling. Because you believe strongly in an idea, innovation happens.
Brain is the power center which moves in mysterious ways. Brain functioning is still one of the biggest and the most mysterious, unsolved riddle, a magic powerhouse, which uses all imaginative and seemingly impossible ways to reach a decision, and then, all of a sudden, for no apparent rhyme or reason discard and dump that same reasoning!
In my considered view, Big Data Analytics is undoubtedly a growing business and an evolving discipline. It presents a fresh, new way to approach an issue or demystify complex consumer behavior. But it's not an exact science and it can't altogether replace, what I like to call 'Gut Science.' Feeling in science is coming back in management with much bigger respect viz. 'Emotional Intelligence.'
Roger Martin in HBR argues that innovators often employ science in their new creations, but real genius lies in their ability to imagine products or processes that never existed before!
Research is cognitive science. But it's proven beyond doubt that the engine of creative synthesis is "associative fluency" viz. the mental ability to connect two concepts that are not usually linked and to forge them into a new idea. That's an act of genius!
That kind of genius, I think comes from gut feeling. Roger explains it very succinctly in his article. If an element cannot be changed in the initial hypothesis, then an executive needs to ask what laws of nature suggest this? If the cause and effect relationship of the hypothesis is very compelling, then data analytics tools may be used to drive choices. But in all other cases, I would rather go with my gut feeling.