IBM data-driven traffic model of Abidjan, Ivory Coast

The data that we produce is collected from social media sites, documents, downloads, email and SMS/text traffic, service and product review sites, internet searches, games, the movements of our increasingly ubiquitous smart phones. The trend for ‘wearables’, devices which transmit data about our exercise regimes, location and behaviour is set to be the next growth market, as a crowd shaping product called Lightwave joins the market; the sleek blue and chrome bracelet is worn by club-goers to collate data on their movement, sound levels and temperature in order to adjust the lighting, environment and track selection, enabling a fully ‘bioreactive’ experience.

Lightwave rave bracelet from Rana June

Does «Big Data» mean anything for Business Intelligence?

As a BI Consultant, my interest is peaked by technological trends linked to practical uses of data driven insights. What’s most intriguing for our community about crowd shaping is that it indicates only the very early stages of a market which seeks to put the consumer at the center of the Big Data model. This is what BI practitioners have been working on, albeit on comparatively ‘small’ data. Just as the true value of BI expertise more often than not resides in asking the right questions to the right people, at the right time, we should take responsibility for asking questions pertaining to the relevance of Big Data, not whether it exists or if we can develop the capacity to use it.

This means that the critical question which will shape the life or death of Big Data is whether it can be used in such a way as to be relevant to the needs of businesses. Simply put, can a business utilise Big Data to make Big Money?

For this reason the question ‘Is Big Data Still Alive?’ continues to be asked. As with any technology trend that carries the implication of hype, Big Data could crash and burn at any time, should the business market collectively decide that the potential gains are not worth the investment to solution it.

One of the challenges underpinning this is that Big Data is all about Big Volumes. In May 2013 an article in the Technology Review stated that a typical American office worker produces 1.8 million megabytes of data per year, or 5000 MB a day. To put that in context, downloading a song on your laptop would consume about 5 MB. While it’s not known how much out of each of our 5000MB a day could be used to foster growth in any given business, it is certain that an industry dedicated to simply storing and processing these volumes of data is required.

Thus, along with the exponential increase in the amount of data points being produced each year, the Big Data economy is rapidly growing. In July 2014 the European Commission outlined a strategy on Big Data which is targeted at increasing investment, knowledge and innovation in the region. The main premise of the strategy is that Europe should be investing to resource an industry with an annual compound growth rate of 40% – ‘about seven times that of the ICT market overall.’.

Valuable opportunities can be found in recruiting and training the next generation of Big Data specialists, hosting Big Data in the Cloud or on-premise, providing marketable insights and trend analytics out of Big Data, technologies for enabling Big Data to be compressed and consumed, streamed and shared, outsourcing of the time and expertise needed to use it and publications dedicated to talking about it. All of this will need to be funded by the businesses seeking to obtain value from that data, and the costs must be relative to the value ultimately gained.

If the Big Data industry cannot demonstrate or quantify that value, then it cannot survive.

If the first consideration of volume is the cost of management and extraction, then the second is that it assumes a degree of homogeneity can be found in the heterogeneity of data in large volumes, which may not be accurate. One of the first open source Big Data insight providers was Google, which in its early forays in 2010 enabled a simple key term search which returned a picture of how many mentions Google could find on that term at a point in time. This gave a simple indication of brand awareness, although it didn’t indicate if the mentions were positive or negative.

Although the technology for detecting trends has moved on immeasurably since that time, the uncertainties in detecting insight from data remains essentially the same. Interest in a particular product does not necessarily mean it will be purchased, and extrapolating the views of the majority of potential customers who produce 5000+ MB a day may not ensure the success of a product launch to your actual customers, today.

The author, journalist and speaker Malcolm Gladwell illustrated the continuity of this dilemma in data driven decision making very effectively in a TedX talk he gave in 2006 entitled ‘Choice, happiness and spaghetti sauce.’

He cited the example of Howard Moskowitz, who went against the trend of assumed homogeneity in data analytics in the 80s by successfully enabling Prego to exploit a market for different varieties of spaghetti sauce. The idea, revolutionary at the time, was that, although it is easy to pull indications of universal ideas from large data sets, doing so could miss the opportunity to target a cluster market which resides below the majority. Now clustering is an accepted approach to data analysis; however it is impossible to know what other opportunities we may be missing, if only we could identify the right questions to pose at the data, or the right data points to collect. And with the Big Volumes of Big Data growing daily, this confrontation threatens to drown any business who takes it on.

Big Data brings another aspect to our Big Uncertainty about the way business will utilise data in the future. However, in many respects the ultimate challenge posed by Big Data is not dissimilar from every other BI question we have already addressed. While Big Data was initially regarded as a test of engineering prowess, once that had been solved, the expertise will lie in identifying and extracting value, or business insight, from the evolving mess of information that we collectively have created.

To do that requires being very smart about designing the questions, which must be shaped by an identified business need, a clear understanding of the consumer or audience and a great degree of responsiveness to change. To achieve that, businesses will have to start by looking at their BI.

In conclusion, is «Big Data» dead?

This simple infographic that we put together may help illustrate the state of Big Data: