The availability of new big data platforms and technologies is enabling organizations and their data scientists to analyze greater varieties of data, in greater volumes and with greater velocity. However, the goal of these advanced data analytics efforts is fundamentally aimed at realizing the same objective that executives and data analysts have been pursuing for decades – how do we gain business insight and answers to our most critical questions, and how do we get these answers so that we can respond to opportunities more rapidly than our competitors?
The metric which describes the ability of an organization to obtain timely answers to inform critical decisions is commonly referred to as “time-to-answer” or “data-to-decision.” The ability of an organization to employ data analytics techniques and processes to accelerate “time-to-answer” and “data-to-decision” represents one of the biggest and the most quantifiable opportunities presented by the advent of new “big data” platforms and technologies.
Time-to-Answer = Time-Value-of-Money
The ability to organize, digest and navigate through vast quantities of disaggregated or unstructured data is a requirement to identify the critical insights and address the most important questions that organizations must answer to make key decisions. The processes of data aggregation, data preparation and data analysis are time consuming, costly and fraught with error, misunderstanding or miscommunication.
Whether it is analyzing sensor data from satellites for military intelligence of predicting extreme weather patterns, or whether it is analyzing large volumes of customer financial transactions, the time incurred to generate answers results in significant costs in computing, process design and management and human capital. The ability of organizations to shorten this cycle, and to employ techniques and novel approaches to accelerate time-to-answer, will result in significant cost savings.
This is why the ability to accelerate time-to-answer is not only important in accelerating the time in which an organization can move from data-to-decision, but also translates to significant cost savings. Hence, the metric time-to-answer also directly correlates to the time-value-of-money.
One technique that many organizations are successfully employing to accelerate time-to-answer is the establishment and adoption of an “analytic sandbox.” An analytic sandbox is a similar concept to a “center of excellence,” but with a very specific purpose.
Organizations must distinguish between “known” and “unknown” or new data and processes. Whereas, “known” data is commonly managed in the context of rigorously maintained production environments with robust standards and processes, “unknown” data is typically immature, and not at a stage of knowledge that this data can be incorporated or transformed into mature production environments.
This kind of raw, fresh, unknown data is ripe for an “analytic sandbox” environment, where an organization is trying to discern patterns or identify an important insight or element without the benefit of established processes or a formalized understanding of the data. An example from the military intelligence community would be the ability to detect threats from vast streams of communications or sensor tracking, but without the benefit of knowing precisely what one may be looking for.
Similarly, businesses frequently need to make critical decisions without complete data. They need to act upon the “best data available” or “directional” data to make decisions or extend business offers in a timely manner. In these situations, production-based systems are typically too slow and insufficiently responsive to respond to “real-time decision making.”
The analytical sandbox provides a mechanism for organizations to sift through and analyze vast amounts of information, seeking directional patterns. As these patterns become formalized over time, an organization may establish rules and processes, and migrate certain data from the analytic sandbox to a more rigorous production environment, where it is managed with greater controls.
Managing for Big Data
As noted, “big data” is accelerating the ability of organizations to rapidly process and analyze information and accelerate “time-to-answer.” The volume and velocity that characterize big data means that organizations will need to develop continuous processes for analyzing and synthesizing data. The ability to integrate “new” and “known” data will enable organizations to accelerate “time-to-answer” and respond to the opportunities or threats facing them in the immediate moment.