Could big data be hiding big clues about security threats? IBM thinks so, and it announced this week a new Security Intelligence with Big Data initiative.

The initiative combines big data analytics with capabilities to detect and prevent both internal and external threats, a combination that the company said allows for analysis exceeding normal security-related efforts. The custom analysis dissects large amounts of both structured and unstructured data, and includes real-time correlation with security threats and forensic abilities to gather evidence.

Hunting, Not Farming

In its announcement, IBM cited the example of the Depository Trust & Clearing Corporation, or DTCC. The organization is a financial services transaction clearing and settlement provider that helps to protect financial markets and systems on behalf of member firms, and its volume is immense – 3.6 million securities in 122 countries, valued at nearly US$ 40 trillion.

Mark Clancy, Chief Information Security Officer of DTCC, told news media that government and the financial industry need to move “from a world where we ‘farm’ security data and alerts with various prevention and detection tools to a situation where we actively ‘hunt’ for cyber-attacks in our networks.” He said that IBM’s solution provides “a practical way to gain visibility across our environment,” matching real-time security awareness with “meaningful insight into historical activity across years of diverse data.”

The IBM big data/security solution combines its QRadar Security Intelligence Platform with its InfoSphere BigInsights, providing insight harvested from massive amounts of new and historical data. The company said that this approach helps organizations “answer questions they could never ask before, by widening the scope of investigation to new data types.”

High-Speed Querying

The solution, which was developed in IBM Labs, offers real-time correlation and anomaly detection, high-speed querying of security intelligence data, a graphical interface for visualization, forensics and analysis of structured and unstructured data.

Structured data includes alerts from security devices, operating system logs, DNS transactions and network flows, while unstructured data could be emails, social media interactions, full packet information or business transactions.

Some use cases that require such abilities to more flexibly analyze large amount of data, the company said, include advanced persistent threat detection, fraud detection and insider threat analysis.