These Trends are Driving Structured Data Archiving

6 minute read
David Roe avatar

The growing use of Apache Hadoop, increasing data warehouse volume sizes and the accumulation of legacy systems in organizations are all factors fostering structured data growth. To manage it, organizations are being forced to look at data archiving and how to get non essential data out of mainstream enterprise applications.

The findings appear in Gartner’s first Magic Quadrant for Structured Data Archiving. This is the first time that Gartner has produced such a Magic Quadrant, which features 13 vendors in total including the four leaders.

While many enterprises are yet to embrace data migration and archiving, Gartner estimates that migration will represent about 20 percent of all information governance projects. To help enterprises do this, four vendors have emerged as the leaders in the archiving space: HP, IBM, Informatics, and Solix.

In this article, we will explore this quadrant and take a look at emerging trends and the market in general. In a second article, we'll take a deeper look at the leaders and what they are offering. 


A New Need?

Given that many enterprises are currently buckling under the weight of structured data, it is surprising that this is Gartner's first Magic Quadrant for this area. But maybe that’s because the need for this kind of capability has only started to be verbalized recently.

In fact, a constant feature -- and a surprising one -- that we find when looking at information governance is that archiving, in many enterprises, appears accidental rather than part of any coherent strategy. That is rapidly changing, however, as data creation continues to experience massive growth and the need to keep and archive data increases as regulatory regimes become increasingly demanding. 

The amount of information that enterprises are being forced to keep has grown so rapidly and to such volume that by 2016, 75 percent of structured data archiving applications will incorporate support for big data analytics, Gartner reports.

There are a number of other factors that are driving the structured data archiving market, not least of which is the fact that it just doesn’t make sense for enterprises to be running up storage bills by holding on to information that can be removed from direct line of business processes.

Gartner also points out that organizational mergers and acquisitions, data center consolidation, and migration to cloud based applications have accelerated the requirement to retire legacy and redundant applications.

Application retirement presents numerous cost benefits and efficiencies that further fuel this trend, the research says. Summing up, the report concludes:

The benefits of using this technology include reduced capital and operating expenditures, improved information governance, lower risk, and access to secondary data for reporting and analysis."

Structured Data Archiving

Just to be clear about what Gartner researched in this Magic Quadrant, the definition of structured data archiving being used is:

Learning Opportunities

The ability to index, migrate and protect application data in secondary databases or flat files typically located on lower cost storage for policy based retention. It makes data available in context and protects it in the event of litigation or an audit."

It address four issues in particular:

  1. Storage optimization: Reduces the volume of data in production thus reducing capital and operating expenditures. It also provides access to primary and secondary data for reporting and analysis
  2. Governance: Keeps appropriate data for compliance and regulatory issues, particularly structured data related to financial issues.
  3. Cost optimization: Structured data and application retirement can result in significant data costs savings.
  4. Data scalability: The technology can manage large volumes of nontraditional data resulting from newer applications that can generate billions of small objects. Scalability to petabytes of capacity is required in these cases.

Data Archiving Market

Gartner estimates that the market was worth $270 million last year, but also that it is growing at a compound rate of 10 percent every year. There are, however, four trends that will dominate this market over the coming years:

Big Data: The market, however, is changing and expanding due to growth in data, application retirement, information governance and big data analysis opportunities. The use of Hadoop and data warehouses means that organizations can keep more information, as well as social and machine data that doesn’t fit into databases.

Information Governance: Structured data contained in applications, especially legacy applications, is an easy target for external auditors. In particular, they are quick to identify applications and associated lack of controls around the protection of sensitive information. If the data is contained in legacy systems that are so old they are no longer supported, or badly managed, they may expect enterprises to migrate to newer versions or entirely new applications. The preparation and response to e-discovery requests is also a core element in structured data e-discovery and governance.

Application Retirement: Organizational mergers and acquisitions, data center consolidation, and migration to cloud based applications have accelerated the requirement to retire legacy and redundant applications. There are numerous cost benefits and efficiencies that are also pushing this trend. However, enterprises need to understand the relationship between data and applications before they can embark on a retirement program that will cover issues like data access and safe storage. A number of vendors are already providing the tools to help enterprises retire their applications.

SAP and Structured Data: The structured data archiving market includes solutions that archive data from applications such as those from SAP. Like other applications, SAP instances need to be retired with data managed systematically for cost and governance reasons. Vendors like IBM and Informatics provide solutions for archiving directly from databases, as well as active archiving and application retirement for SAP and other ERP and CRM applications. As part of the Magic Quadrant for Structured Data Archiving and Application Retirement, Gartner evaluated and identified SAP archiving and retirement solutions.

This year, which is the first year this MQ has been published, there are 13 vendors that made it into the mix, but it doesn’t mean that there aren’t others who offer interesting products. There are also a number of vendors that Gartner thought it worth mentioning even though they didn’t make it into the Quadrant. Among them:

  • Actifio: It comes with a copy data virtualization platform called Copy Data Storage and offers long term retention of backup data that can be actively used and repurposed. It also supports databases such as MS SQL, as well as business applications such as SAP and Oracle E-Business Suite.
  • CommVault: It offers a single platform approach to backup and archiving as well as supporting data and document archiving for SAP modules.
  • Gimmal: Using its ERP-Link product, takes the approach of enhancing and using Microsoft SharePoint as a strategic enterprise repository for managing content, including SAP data and documents.
  • SAP:  NetWeaver Information Lifecycle Management, which archives SAP data and provides retention management capabilities.

Next week, we'll look in more detail at the vendors that made it into the Leaders’ Quadrant.