A surprisingly large percentage – 89% -- of Hadoop users are not certain whether native security tools provide enough protection for their Big Data projects. The irony is almost the same percentage of users, 86%, said that data security is a critical requirement for their Hadoop data lake or hub.

The survey was conducted by Protegrity at last month's Strata + Hadoop World Summit in San Jose, Calif.

Despite the giant question mark over Hadoop's security creds, users showed every indication of continuing to deploy the technology. Some 80% of respondents said that their organizations are using Hadoop in production environments; 80% of respondents also said their organizations will be spending more on Hadoop-related projects this year.


Why the Disconnect?


There are some logical reasons for this disconnect between Hadoop implementation and users' understanding of its security, Suni Munshani, CEO of Protegrity, tells CMSWire.com.


The main one is that a lot of Big Data projects are still in the pilot or Proof of Concept stage, he says. "These projects are not integrated into the larger security environment yet."


Another reason for the gap between users' acute awareness that security is essential and their haziness of whether Hadoop is really up for the job is that vendors are telling clients that security is part of the specs.


It's not that they are lying; rather it is a different variation of the aforementioned problem. The vendors really cannot answer questions about security until they understand the full scope of the project and how it fits into the larger ecosystem.


"Right now the question of security is very much a 'check off the box,' situation," Munshani says. "Security is asked about and the vendor gives a stock answer of 'yes we provide security.'"

Good Security, Big Data Don’t Mix Well

What is worrisome to managers watching these issues unfold is that good security is at odds with Big Data, Munshani says. "These projects require data to be made available in multiple environments at very high speeds and that is where it gets complicated."