In Pursuit of Search Clarity

One of the rarely mentioned benefits of writing a book is finding out how little you know.

After writing almost 100,000 words for the second edition of Enterprise Search for O’Reilly Media and looking at the comments a small group of search experts made on the draft, I learned that some sections were difficult to read and some basic facts were wrong or badly scrambled. This edition will be twice the size of the first edition, which was published in 2012.

The main reason for this increase was to respond to requests for various topics from people who wanted as comprehensive a book as possible. Handling these requests has taken me into areas which -- until now -- had been at the edges of my domain knowledge. I hope that this is not too visible when the book emerges in a few months’ time.

Search lies at the intersection of language and technology, which makes it a challenge to translate the technology into reasonably clear English. Elasticsearch's Clinton Gormley and Zachary Tong have set a high standard for clarity with their recent book on Elasticsearch.

The book goes beyond a do-it-yourself guide on Elasticsearch implementation to describe -- with great clarity -- each individual element of search technology along with the relevant code. But this takes over 700 pages to accomplish. Search is complicated technology and I remain impressed by Peter Morville’s elegant book on search patterns in which he explores aspects of user interface design without once mentioning the technology.

Search 101

This challenge came up with one of my clients. How do you translate search technology into something understandable for not only business managers but IT staff who lack previous search experience?

The IT team had developed a search strategy that didn't stack up. While most of the individual elements were fine, the proposed strategy combined the elements in a strange way. Trying to explain to both the IT team and the business team why the strategy might not be ideal was very difficult, even with a great deal of patience on both sides.

The fundamental problem was that in order to explain my concerns, I had to provide a fast account of the basic principles of search over a telephone conference call. In retrospect a Webex-type call might have been better, but the need for the explanation only became clear once we started the call.

To most people search is akin to magic, at least when it works well. Other times it probably seems more like malevolent black magic at work, displaying documents that seem miles away from the query.

As a former chemist, I know the power of using diagrams to convey complex chemical reactions. I'd do this in a space the size of an index card (assuming you can remember an index card's size) using a common graphics language for chemical structures.

This made me think about developing a set of PowerPoint slides to explain in process flow terms the way search works and how the way it works impacts search performance and search satisfaction.

It seemed so easy ....

So in parallel with completing the book, I have also been developing this slide deck. The process has helped to validate the book's structure, though quite a bit of work remains.

The next edition of Enterprise Search will have its own website. The aim is to create a reasonably comprehensive resource for enterprise search and enable me to take the search content away from the Intranet Focus site. The slide deck will be one of the downloadable items from the site, in the hope that others may find it useful for presentations.

If any reader has, or is aware of, a similar search slide deck I’d be interested in sharing approaches.

Getting the Message Across

What I'm aiming to do with both the book and the slides is to bridge the gap between information retrieval and enterprise search. It still surprises me that my book has not had more competition (though that situation has been good for my royalty checks).

I’m aware that some people may hesitate from taking action on search out of fear of getting into difficult technical areas, and in a way am concerned that the size of the book might be counter-productive because of this. Other enterprise applications are essentially databases, but businesses require no understanding of database technology to get the best out of them.

Search is different. You need to understand the technology to realize the problem and the solution are all about content quality. 

Creative Commons Creative Commons Attribution 2.0 Generic License   by  rawdonfox