Will Big Data Solve Everything?

Is big data analytics the holy grail?

As I think more and more about big data analytics, I wonder if we will really find the answers we are looking for in that mass of data. This whole thing is kind of reminding me of the search for the Higgs boson particle – we all have this sense that the answers must be somewhere in that mass of big data that we are collecting – and if we simply collect the right information and analyze it the right way at the right time then maybe we will have all the answers.

The answers out of big data are turning it into some kind of “God Data” – in the same way the Higgs particle was called the God Particle – it could contain all the answers to the question of life the universe and everything. If we just ask the right question we’ll get the answer to everything.

The question is – will be end up with the answer, or just more questions? Maybe, as Douglas Adams put it – we may get the right answer, but we never asked the right question.

Maybe its a human nature thing – when confronted by a giant repository of information, our initial instinct (no matter how incorrect) is to believe that everything we are looking for is there, we just need to figure out how to find it. Just like those who use Google for all of their research have this impression that Google is attempting to present the most relevant information to you based on your search query. Having worked for a similar company in the past, I can tell you that relevance may not be the major factor in the list of results that you get – yes – even barring the ads. Don’t you wonder sometimes why its takes a lot of manual mining to find what you are looking for, despite the engine supposedly refining its search based on your search history – you nearly never get the most relevant stuff up front?

But I digress. My sense is that the reality is that big data will give us some of the answers but not all of them, and possibly not the ones we want. Why is this? Simply because no matter how much data we currently collect, its still not enough for us to read the users tea leaves and truly understand what they are looking for.

Not to say that that day isn’t coming. Once every device is studded with sensors and begins to report our every action, keystroke, location, movement, heart rate etc, we may then be able to see the patterns. There is currently, and from what I can see, no service which can provide the most relevant info (yes, even Google Now) when you need it – and its really only because it still doesn’t know enough. Glass, with its ability to record everything you see and your location, might help get us there, however privacy concerns might kill this source of data.

My sense is that answers will come – but only when we hit some kind of tipping point with the data – that we finally have enough in order to answer the questions – and when we know how to ask the right ones. That day may be closer than we think.