In September I was taking part in a European Commission conference on search computing at which a senior executive of a major search technology vendor proclaimed that enterprise search is dead, and that the focus should be on data analytics. Without doubt there are some significant benefits in enabling managers to look for trends and outliers in the mass of data that is now being collected by many companies, but that does not mean to say that "enterprise search is dead."
Over the last couple of years more than a hundred books and countless reports have highlighted the benefits and challenges of providing effective access to Big Data. The October 2012 issue of Harvard Business Review has three articles on the topic, so it must be important! This still does not negate the importance of enterprise search.
Enterprise Search and Big Data
Most enterprise-level applications support defined business processes and associated workflows. As a result the manager responsible for the workflow (e.g. customer relationship management) is also willing to take on budget responsibility and work with IT on solution development. Enterprise search does not support specific workflows; it supports all of them. This makes it very difficult to find someone who wants to take responsibility for enterprise search management and also very difficult to agree who has budget responsibility.
For some time there has been a view that the unstructured information in a company represents 80 percent of the volume of information, a view heavily promoted by Autonomy. The evidence is anecdotal but even if the balance is in fact 50-50 the result is that Big Data applications are only going to meet half the business requirements for information and data access.
There are also vast collections of project reports, service reports, business plans, patient records, etc. where being able to find relevant information by using either keywords or through a natural language enquiry in whatever are relevant languages is absolutely business critical.
If you want to get a sense of the scale of the mismatch between business requirements and the availability of enterprise solutions then have look at surveys carried out by MarkLogic in 2011 and 2012, Findwise, Oracle and SmartLogic:
- MarkLogic: MarkLogic Survey Reveals Unstructured Information is Growing Rapidly, will Soon Surpass Relational Data
- MarkLogic: Big Data is Real and it is Here: 2012 Survey on Managing Big and Unstructured Data
- Findwise: The Enterprise Search and Findability Report 2012 is ready
- Oracle: From Overload to Impact: An Industry Scorecard on Big Data Business Challenges
- Smartlogic: Mind the Enterprise Search Gap: Smartlogic Sponsor MindMetre Research Report
Although some of these surveys are notionally about Big Data you will see that the need to search unstructured information is also seen as very important as this category of information provides invaluable context for any analysis of data. This is highlighted in a recent AIIM survey in which over 60 percent of respondents noted they would find it very useful to be able to correlate text-based data with transactional data, but only 2 percent are able to do so at present.
What is now emerging is a view of enterprise search as a managed framework of search applications which enable employees to find the data, information and knowledge that they have the authority to access in order to make business decisions. This is a view that is now being promoted by HP, IBM and Oracle after high profile acquisitions last year, and by the open source search community (notably Lucid Works) where Lucene and Hadoop are well positioned to provide enterprise-scale solutions to information and data management.
The Skills Gap
The wider adoption of enterprise search is not just a technology issue. There are two major barriers to gaining the best from the technology. The first of these is an understanding of how the technology works and what it is capable of delivering. Although there are many books on the science of information retrieval and blogs from search vendors, these do not help IT managers and developers understand the complex world of search and realize that Google is not necessarily the answer.