Reality is beginning to bite the Internet of Things (IoT). After months of enthusiastic discussion about the opportunities it will provide and how much it will be worth, many of those looking to play in the IoT space are starting to look at the potential problems, including data management.
Though everyone knows managing data will be a problem once the IoT is up and running at full scale, few have really considered the potential data storage problems.
The Problem With Data
Sure there have been hypothetical discussions around compliance, privacy or the kind of information that consumers will be happy to offer to businesses in exchange for better customer experiences. But there has been little discussion around the subject of where exactly enterprises plan to store the massive amounts of data that will be created.
Think about it. According to research from Gartner, there will be an estimated 26 billion units installed globally by 2020 with many more on the way in succeeding years as the price of processors drops. In the near future, it will be feasible to install a processor into just about everything.
Where is all the data provided by those processors going to be stored and what are the problems around them?
This is not just a brainteaser. It is a very practical and real problem. After all, if enterprises are to get the bountiful insights into customer activity like the IoT promises, they are also going to have to keep all that information somewhere while it is being analyzed.
In a recently published paper from Garter entitled The Impact of the Internet of Things on Data Centers, Gartner identifies the principal issues that are going to have to be resolved before enterprises can start to benefit from the IoT. Fabrizio Biscotti, research director at Gartner, summarized the problem as follows:
IoT deployments will generate large quantities of data that need to be processed and analyzed in real time. Processing large quantities of IoT data in real time will increase as a proportion of workloads of data centers, leaving providers facing new security, capacity and analytics challenges.”
Connecting Remote Assets
The problem lies in the nature of the IoT itself. It will connect remote devices and systems and provide a data stream between devices and decentralized management systems. The data or even the devices will be incorporated into existing organizational processes to provide information on the location, status, activity and functionality of those systems, as well as information about the people who own and operate them.
The amount and type of information differs than other sets of big data that comes from social media, for example, in the following ways:
- It tends to arrive as a steady stream and at a steady pace, although it can arrive in batches like test logs that can be processed and passed on straight away
- It comes in very large quantities and accumulates very fast
- The real value can only be uncovered using analytics
- It is rarely used for production purposes
- It is deleted very quickly, unless it is needed for compliance reasons
The IoT Data Challenge
The technologies to address the big data challenge already exist, like Hadoop or NoSQL, providing horizontal scalability, high capacity and parallel processing at prices that make them affordable and economical.
For the moment, though, IT departments in enterprises have not had to deal with IoT data as a unique dataset in its own right. For the moment at least, the first sets of what will make up IoT data are arriving in the storage layer in the same way other unstructured data does.
The result is that traditional storage architecture and management software can treat IoT data the same way as they treat other unstructured data.
However, this is all changing rapidly. With the development of wearables for consumers and the emerging use of smart machines the portion of IoT as a subset of big data will grow quickly forcing enterprises to think their infrastructure to enable scalability and to make them cost effective.
With these changes come seven different challenges that enterprises, and in particular IT departments will have to manage:
The enormous number of devices, coupled with the sheer volume, velocity and structure of IoT data, creates challenges, particularly in the areas of security, data, storage management, servers and the data center network, as real-time business processes are at stake," Joe Skorupa, Gartner vice president said.
Gartner has identified several challenges:
If the digitalization and automation of millions of devices will create a whole new security landscape as enterprises attempt to protect themselves, it will also create new opportunities for operational technology security providers.