Big Data: Use It Or Lose It

Use It Or Lose It

Resources like oil, water and good ideas may be in short supply, but the same can’t be said for data. Businesses of all sizes face a deluge of it on a daily basis with volumes rising exponentially and, it would seem, no end in sight. Collecting all this data isn’t an issue – the IT industry has made sure of that – and we’re also coming to grips with analysing and extracting value from this mass of raw information. All the more so as companies start to take advantage of Big Data applications. But it’s not all good news.

On the plus side, Big Data applications make it possible to identify useful patterns and trends that might, otherwise, have gone unrecognised. However, the majority are designed to sift through and analyse information sometime after collection. Fine if you’re looking for long-term trends, but not if you want to take more immediate advantage of collected data, much of which comes with an implicit “use by” date, after which its value will quickly diminish.

This data, with its implicit ‘use by” date, needs to be analysed, processed and used in a very short window of time. A unified in-memory solution is necessary together with the ability to analyse events in real-time to process all data (collected data and data in motion) and take action based on the “use by” date.

Let’s look at a retail example, where there’s a lot to be gained from identifying potential customers on the basis of historical purchases, browsing habits and so on. The big high street retailers all understand this, but historical data is, by its very nature, “cold” data and retailers still have to invest in traditional marketing campaigns to tempt customers back into the store or shop online.

What if you could, instead, tie in that historical analysis with “hot” information such as real-time location data from mobile devices, inventory statuses, available offers, and other “use by” data assets? It then becomes possible to identify potential customers while they are shopping and calculate the probability that they are ready for an offer.

We have given this type of analysis a name – the “Psychological Router” and, based on what it tells companies, customers can be sent personalised offers, buying suggestions and other information direct to their phones, potentially gaining extra value from the information base.

Or what about the utilities industry? Regulatory compliance and environmental sustainability force utility companies to increase their operational efficiency by monitoring their activities in real-time across the entire ecosystem. Again, the data collected needs to be analysed while reasonably “hot” to optimise demand response, load management and other activities to anticipate and respond to faults and outages.

Similarly, while patient data can be used by healthcare researchers in the long term to find causes and cures for diseases there’s even greater value to be had from more immediate analysis. Real-time analysis of monitoring devices, for example, can highlight the need for immediate medical attention, changes in treatment and other action. The “use by” date for such data can be incredibly short and, once passed, Big Data analysis is no longer of any use.

Inevitably there will be applications where immediacy is not so important but, equally, numerous other examples where fast data analysis can add value to Big Data applications. We just need to understand the distinction and, where real-time analysis can offer up additional benefits, make sure we use it while the data’s hot.

As EMEA CTO for TIBCO Software, Maurizio is responsible for ensuring consistency of technology vision, providing strategic sales support and support for AR, PR, marketing and M&A initiatives, as well as serve as a conduit for the Global CTO function. Maurizio has more than 25 years of experience in IT, working for several software vendors, such as TIBCO Software, IBM, Siebel, SOA Software and Red Hat.