Earlier this year I was sitting next to an enterprise search manager, looking through their vast collection of search logs. All of them were quantitative, derived from clicks or non-clicks. The organization had been striving to improve the relevancy of search results so as many users as possible would see the most relevant results on the first page, but their efforts didn't result in more clicks on the relevant items. The search manager asked me, "What are we doing wrong?"
Understanding SERP Use
I responded that the organization wasn't focusing enough on the user behavior with enterprise search and in particular, how people were using the search engine results pages (SERP). As we looked at search terms with a significant number of results it became obvious the documents appearing in the first few SERPS were ones employees would already have, so there was no reason for them to click on them. At first this seems counterintuitive, but an important use case for search is to check if something was missed in an earlier search. The user might even go through a dozen SERPs without clicking and end, feeling satisfied they had found all the relevant information. The result is a highly satisfied user and a worried search manager, who believes all of their careful work on improving relevance is having no impact.
Related Article: How Sticky Is Your Enterprise Search?
Stopping Strategies: Why People Stop Searching
I have mentioned the topic of stopping strategies before in this column. A stopping strategy is how users decide when to stop looking at search results pages. Among the earliest research into the topic is a paper published in 1995 by Kathryn Nickles and her colleagues, but it is only comparatively recently that the value of understanding stopping strategies has been recognized. My return to the topic was prompted in part by the release of David Maxwell's (University of Glasgow) superb PhD thesis on the subject. It is over 400 pages long, which should give you an indication of the complexity of the subject.
Maxwell discusses a range of search stopping strategies. A common one that he did not include in his thesis is when a search user expects to see a specific document listed on the first page of results. Seeing it high up on the results page will validate their search strategy, but again, they will not click on it. If the document does not appear, they might look on a few more pages, but if the anticipated document fails to materialize, they may well decide the search engine is rubbish. While there could be many good reasons why this document is not listed, the user will not be aware of them, or care. When optimizing search and employee productivity, plotting pages reviewed against personas and a range of queries can be a very powerful analytics tool.
How Long Is a Session?
Deciding what counts as a "session" is another challenge in enterprise search management. In an ideal world, a person will try a different query if their first search is unsuccessful. A Microsoft Research paper provides some statistics on session lengths and how people manage search sessions, though this is specific to the web.
In enterprise search, people are under time constraints and may not try a revised approach until later in the day or even the next day. Search analytics has to be able to support these extended sessions to facilitate query modification. Click logs also need to be able to differentiate between a new query and a revised query. While this is very subjective, it can bias the search logs. (This leads into the very interesting and complex topic of how searchers expand and refine queries, but that is for next month’s column!)
Related Article: 4 Dimensions of Enterprise Search Success
Be Careful What You Count
In both web search and ecommerce, search relevance depth (e.g. precision at the 10th result) can be a good metric of search performance. Trying to improve this for enterprise search may compromise the recall performance, something that has little value in either web or ecommerce search. Most of the standard information retrieval metrics require using test collections, which can be challenging to construct and manage in an enterprise search environment.
As we saw in the first example, relying on click logs to improve enterprise search performance can result in a decrease in search use and satisfaction. Understanding stopping strategies can help, but there's no substitute for undertaking real-time usability tests and having face-to-face discussions with groups of search users. When did you last attend a departmental or project team meeting to understand their search requirements, expectations and levels of satisfaction?