In September I was taking part in a European Commission conference on search computing at which a senior executive of a major search technology vendor proclaimed that enterprise search is dead, and that the focus should be on data analytics. Without doubt there are some significant benefits in enabling managers to look for trends and outliers in the mass of data that is now being collected by many companies, but that does not mean to say that "enterprise search is dead."
Over the last couple of years more than a hundred books and countless reports have highlighted the benefits and challenges of providing effective access to Big Data. The October 2012 issue of Harvard Business Review has three articles on the topic, so it must be important! This still does not negate the importance of enterprise search.
Enterprise Search and Big Data
Most enterprise-level applications support defined business processes and associated workflows. As a result the manager responsible for the workflow (e.g. customer relationship management) is also willing to take on budget responsibility and work with IT on solution development. Enterprise search does not support specific workflows; it supports all of them. This makes it very difficult to find someone who wants to take responsibility for enterprise search management and also very difficult to agree who has budget responsibility.
For some time there has been a view that the unstructured information in a company represents 80 percent of the volume of information, a view heavily promoted by Autonomy. The evidence is anecdotal but even if the balance is in fact 50-50 the result is that Big Data applications are only going to meet half the business requirements for information and data access.
There are also vast collections of project reports, service reports, business plans, patient records, etc. where being able to find relevant information by using either keywords or through a natural language enquiry in whatever are relevant languages is absolutely business critical.
If you want to get a sense of the scale of the mismatch between business requirements and the availability of enterprise solutions then have look at surveys carried out by MarkLogic in 2011 and 2012, Findwise, Oracle and SmartLogic:
- MarkLogic: MarkLogic Survey Reveals Unstructured Information is Growing Rapidly, will Soon Surpass Relational Data
- MarkLogic: Big Data is Real and it is Here: 2012 Survey on Managing Big and Unstructured Data
- Findwise: The Enterprise Search and Findability Report 2012 is ready
- Oracle: From Overload to Impact: An Industry Scorecard on Big Data Business Challenges
- Smartlogic: Mind the Enterprise Search Gap: Smartlogic Sponsor MindMetre Research Report