2014-08-July-Hide-and-Seek.jpgPeople are dissatisfied with the search applications they are using. This is the result -- without exception -- from surveys of search performance. Search managers charged with improving satisfaction levels face the problem of defining what "satisfaction" actually means. Search performance has to be evaluated on three criteria: technical performance, retrieval performance and impact performance, but it's impossible to bring all three together in some mathematical formula for "satisfaction."

Technical Performance

Google and Bing have set the expectation that results will be returned in less than a second, but both companies have invested billions of dollars to achieve this. Many corporate websites use a hosted search application which can take much longer. The Shell corporate website currently takes around 10 seconds to respond to a query.

The challenges are much greater internally. Security trimming can impact search response times, as can federated search implementations. There is more to technical performance than minimizing query response times -- attention needs to be paid to crawl and indexing speeds and the speed with which the main search index is updated.

Retrieval Performance

Now we are in the area of relevance, recall and precision. Relevance is very subjective. Two users with similar skills and roles and even office locations, may take a very different view on the relevance of a set of results. This may increasingly be the case with mobile search where consumer applications set the expectation that the search will be refined based on location.

Recall is a measure of the percentage of relevant results returned as a percentage of all relevant results in the collection. In most cases there is no way of knowing how many relevant results are in the collection so this measure has significant limitations, especially with very large collections of documents. At least one major law firm has over 1 billion documents. When a user is trying to achieve high recall they will use very high level search terms, such as "drill," and then either use filters and facets to narrow down the search scope or take an entirely different query route based on a review of the initial set of results.