The era of Big Data brings both challenges and opportunities

A a CIO you are,essentially, in charge of data, the company’s crown jewels. You need to protect it from unauthorised access, ensure that it is accessible to those who have a right to access it, guarantee that the systems that contain it attain as near a 100% up time as possible and generally make certain that the infrastructure for that data is robust.

It is fair to say that in the past that has been manageable but there has been a tipping point where companies are now creating and storing more data than we ever thought possible five years ago. We are now in the era of Big Data and a new report from McKinsey shows there are both challenges and opportunities.

Up until recently your organisations data was created by and large on desktop computers contained within the concrete, steel and glass safety of the office building. But with smartphones tablet computers, netbooks and even the old PDA style computers that is no longer the case.

Data can be created, accessed and stored from anywhere and with companies now generation gadzillionbytes of data on its customers, employees, other organisations and so on, there is a distrust of what this information will be used for. But the McKinsey report states categorically that once we put the myth and potential for data misuse to one side there are phenomenal benefits.

The report “Big data: The next frontier for innovation, competition, and productivity” explores the state of digital data, how different domains can use large data sets to create value, and the implications for the leaders of private-sector companies and public-sector organizations, as well as for policy makers. The report’s analysis is supplemented by a detailed examination of five domains—health care, retailing, the public sector, manufacturing, and personal-location data.

McKinsey states that in the United States alone there is a shortage of between 140,000 and 190,000 with the relevant expertise to handle this data and an estimated 1.5 million managers and analysts to understand and make decisions based on this new Big Data. My worry is if the country is that short now how can it create a viable and capable Big Data Management Structure without playing catch up?

Despite this “minor hiccough” in the great scheme of things the report is remarkably upbeat about the future benefits, stating:

1. Make big data more accessible and timely. Transparency, enabled by big data, can unlock a great deal of value. In the public sector, increasing access to data across separate departments can sharply reduce search and processing times. In manufacturing, integrating data from R&D, engineering, and manufacturing units to facilitate concurrent engineering can cut time to market.

2. Use data and experiments to expose variability and raise performance. As organisations create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to sick days.

3. Segment populations to customise. Big data allow organisations to create ever-narrowing segmentations and to tailor services precisely to meet customer needs. This approach is well known in marketing and risk management but can be revolutionary in areas such as the public sector.

4. Use automated algorithms to replace and support human decision making. Sophisticated analytics can substantially improve decision making, minimise risks, and unearth valuable insights that would otherwise remain hidden. Such analytics have applications from tax agencies to retailers.

5. Innovate with new business models, products, and services. To improve the development of next-generation offerings and to create innovative after-sales services, manufacturers are leveraging data obtained from the use of products. The emergence of real-time location data has created a new set of location-based mobile services from navigation to people tracking.

Kevin Tea is a journalist and marketing communications professional who has worked for some of the leading blue chip companies in the UK and Europe. In the 1990s he became interested in how emerging Internet-based technologies could change the way that people worked and became an administrator on the Telework Europa Forum on CompuServe. With other colleagues he took part in a four year European Commission sponsored project to look at the way that the Internet could benefit remote communities. His blog is a resource for SMEs who want to use cloud computing and Web 2.0 technologies.