It is easy to become despondent about the state of enterprise search. Only 11 percent of organizations reported very high satisfaction with search in the latest The Digital Workplace in the Connected Organization report, which shows there has been little change over the last eight years. However, from April 29-30, while I was acting as the Chairman of the Enterprise Search Europe 2014 conference in London, despondency was certainly not on the agenda.
Over 100 delegates from 13 countries were in the main venue to hear Dale Roberts start the conference with reflections on the need to make better decisions. He focused on the role of searches in providing access to the wealth of information that organizations have invested in over the years and the importance of delivering information to the person making the decision. The business case is very persuasive.
Coincidentally, an independent report on the commercial problems affecting the Cooperative Bank in the UK was published during the conference. To quote from the report:
Effective boards need clear and relevant information, provided in a timely fashion. This is particularly the case when those receiving the information have little experience on which to rely in understanding what is important. Board papers examined during the course of the Review were frequently long on detail. But they sometimes omitted important information (or included it, but not in a prominent way), failed adequately to draw out key facts or gave an overly-optimistic picture of what was really happening.” (p. 136)
Big Data Plus Search
Many of the papers emphasized the importance of seeing big data and search as complementary technologies, a point very well made by Iain Fletcher of Search Technologies. The approach used is to combine caching and a Hadoop database to improve the overall performance of a search application when dealing with very large quantities of databased information. Ronald Hobbs of Reed Business International described a project requiring access to 100 million documents, 72 business units and up to 190 queries per second. Reed had originally used FAST ESP engine but they moved to an Apache Solr system, replacing the FAST processing pipeline with Search Technologies Aspire project.
Lest it be thought that the world of search is now totally open source, the general consensus was that a paper by Nick Brown, AstraZeneca, was the best of the conference. In the space of around four months a team from AstraZeneca had developed a portal for the research and development teams at AZ which not only searched through the vast internal collections of information held by any pharma company but also tracked developments published in external research papers. The portal used the Sinequa search software and was especially notable for the expertise location application that had been developed, as well as for the use of visualization in the presentation of query results. There is a version of the paper from earlier this year on SlideShare.
The Future of Search
An innovation this year was a duet for two keynotes. Jeff Fried (BA-Insight) and Professor Elaine Toms (Information School, University of Sheffield) joined together to bring perspectives from enterprise search and from information retrieval on the development of search in the future. Jeff made the point that the total cost of ownership had dropped by an order of magnitude over the last few years, but this was perhaps not obvious to managers considering investment in search. It was interesting to see areas of agreement and disagreement between the two speakers, but there was certainly a consensus that we were moving towards providing users with highly customized versions of search applications that enabled them to meet specific needs or make specific decisions, rather than trying to develop one single enterprise search application that was then federated across multiple repositories and applications. The "futures" theme was continued with presentations on text mining, collaborative information seeking and multimedia search.
And in Other Tracks …
The conference started with two consecutive workshops on SharePoint 2013 search and, in addition, there was a conference track. Compared to similar workshops and presentations last year it was clear that there was much more experience now available on how to get the best from SharePoint 2013. Solutions have been found to a number of issues with SP2013, but the downside is that there are some quite complex issues -- especially around security management and distributed indexing -- which still need very careful attention.
One of the themes that emerged from many of the papers was the need to ensure that content quality was as high as possible. Many search applications fail because IT departments are not aware of the inadequacies of the content that is being indexed.
Despite an Underground strike, attendance at the final panel session was high, as was the enthusiasm for the potential of search as an enterprise application. However, there is still a significant need to get the message across to senior management teams who seem to have a limited appreciation of how search can transform business potential.