A year ago I suggested that search would become business critical in 2013, and I think my forecast was in line with reality. Certainly the Findwise 2013 Search and Findability Report indicated that more organizations were working on a search strategy in 2013, and that compared with the 2012 report, businesses displayed a greater degree of commitment to getting the best out of investment in search applications.
But as the methodologies of the 2012 and 2013 surveys were a little different, it is probably too soon to be confident about trends. Let's take a look at some of the noteworthy developments in 2013 and I'll offer a forecast for 2014.
SharePoint 2013 Search
A substantial number of organizations moved closer to SharePoint 2013 implementation during the course of 2013. They found that although the search functionality of SharePoint 2013 is significantly better than in SharePoint 2010, some issues still exist that require special attention.
While the facility to customize search for groups of users is a welcome innovation, it requires a very good knowledge of user requirements and a well-skilled search team. This is especially the case when moving up from Search Server for SharePoint in the Standard CAL version of SharePoint 2010, and is not an easy move even for organizations used to FAST Search Server for SharePoint.
A less-welcome innovation: the support for crawling, indexing and presenting search results from content that is not being managed within SharePoint 2013 is poor, especially in terms of providing a consistent user interface to both SharePoint and non-SharePoint content. This means that SharePoint 2013 is not an "enterprise" application, in terms of being able to search across multiple repositories and applications in the way that FAST ESP could be used. 2013 marked the end of full support for FAST ESP and Microsoft seems not to be addressing the wider search market.
For these and other reasons the advent of SharePoint 2013 is causing organizations of all sizes to take a considered look at their requirements for search.
The big data bandwagon is starting to take on a degree of rationality, recognizing that there is a vital need to integrate structured and unstructured information. An SAP survey published in May 2013 indicated that at least 25 percent of data was unstructured, and in many cases it accounted for more than 50 percent of data.
Unstructured data (a.k.a. text, images and videos) requires different search solutions from those that can cope with "traditional" structured big data. Organizations recognize the need for data scientists to help analyze structured data, but overlook the need for information scientists to do the same for unstructured data.
A survey by the Aberdeen Group indicated that organizations integrating structured and unstructured data were able to gain significant improvements in the quality of the decisions they made. I see this integration as an important trend in 2014 as the challenges of making informed decisions on just the smart analysis of structured data become more obvious.
Open Source Search
Open source search has had a good year. LucidWorks has continued to refine its offerings and ElasticSearch is becoming more widely appreciated as an alternative to the Lucence/Solr stack (though it does have Lucence at its core). In October, Yonik Seeley, the man behind Solr, set up Heliosearch with people formerly on the staff of LucidWorks. In addition many companies, such as Attivio and PolySpot (not forgetting IBM Omnifind) have built proprietary code on top of open source elements.
There is no doubt that open source technologies can now match the performance of commercial search applications at a significantly lower up front cost. For internal enterprise applications that may mean that the IT department sees search as a "develop and handover" application, not realizing that open source applications may need a higher level of search support by both the business and IT in order to maintain development momentum. My guess is that over a three year period the total cost of ownership of open source applications may not be substantially different to a commercial application. I hope that someone will undertake some research on this issue.
New for 2014
I offer two topics that I see becoming increasingly important in 2014. One of these is cross-device search, where a search is initially conducted on a desktop and is continued on a smartphone, and vice-versa. There is a very good paper from Microsoft that sets out some of the issues. The second topic is continuous information seeking, where search tasks are carried out by more than one "searcher," often in support of collaborative working. The book on this topic by Chirag Shah, a member of staff of Rutgers University, is a very good place to start.
Editor's Note: Read more of Martin's thoughts on search in Why All Search Projects Fail