Every request for proposal document I've authored over the years includes a question tucked away in the midst of the usual array of functional requirements: does the vendor have a user group, and if so, is it chaired by a customer or a company representative? If the vendor answers no, I ask how it monitors user requirements for incorporation into future versions of the software.

Search vendors are keen to emphasize they are meeting user demands for cognitive search. What's interesting to me is how they are capturing this demand. I have come across very few organizations who proactively ask employees what they need from the search application. The immediate challenge is employees do not know what they need, either in terms of core functionality or in a user interface.

In 2018 Tony Russell-Rose, Jon Chamberlain and Leif Azzopardi published a landmark paper entitled, "Information retrieval in the workplace: A comparison of professional search practices." The team asked experienced searchers from the legal, health care, patent and recruitment sectors about their search requirements, including the extent to which they used almost 20 different elements of a user interface. For the purposes of this column, the differences are unimportant, rather I include the paper to show the wide range of commonly used UI elements in these sectors.

My experience is even if an organization carries out laboratory-style usability tests, it is only done at the initial implementation, not on a regular basis. Invariably the search team has no direct evidence of user requirements, just a collection of click logs. And you can be 100% certain that no search team has wandered the halls of an organization asking if employees want a cognitive search application.

Related Article: Diagnosing Enterprise Search Failures

Playing Telephone With User Requirements

I read the The Forrester Wave: Cognitive Search, Q2 2019 report recently. What immediately struck me was the report airbrushed the role of implementation partners right out. For example Mindbreeze, which was included in the report, will be involved for major contracts, but for smaller contracts the company has a wide range of implementation partners. This is not mentioned anywhere in the report.

The performance of a search application is a function not only of the software's capabilities, but also how well the selected integration partner has performed and the expectation of the budget holder in the customer. Very few search vendors provide global implementation services. The knowledge about user reactions to the software may be gathered to some extent by the implementation partner, but this knowledge will only remain current for a few months. So in reality, the vendor may be relying on feedback from its integration partner and conversations with the internal IT team.

Related Article: Search Won't Improve Until We Understand Why People Search, Not Just How

Straight From the Horse's Mouth: Vendor-User Conferences

We saw a rapid growth of text retrieval software written for mini-computers in the 1980s. Vendors were keen to bring users together to share experiences with each other and the development teams. The STATUS conference was probably one of the best examples of these. 

Learning Opportunities

A feature of the adventurous spirit at FAST Search and Transfer was its very successful user conferences, which used to attract over 1000 delegates. (Interestingly at the 2007 conference, issues about the impact of content quality on search were raised.) The event disappeared following its acquisition by Microsoft. Before anyone points out the many Microsoft conferences — those are for product promotion and are expertly “managed.” If you want to see a model for independent user groups, take a look at the SAP groups.

Sadly, search vendor user group events are a rarity. An exception is Lucidworks, which holds a yearly conference, as well as smaller meet ups around the world. These meet ups are listed on a separate website, not on the main Lucidworks site. Although more than a search application, the effort that HighQ put into its annual HighQ Forum events is exemplary and is a strong selling point for the company.

Related Article: When it Comes to Intelligent Search, Don't Expect Magic

How Do You Communicate With Your Search Vendor?

User group meetings are an opportunity to network with other customers and with vendors, and if HighQ can do it, so can your vendor. 

Ask yourself: how open is your vendor to listening and learning from your experience? If it doesn't have an established channel then consider whether it really cares about its customers. Without such a channel, how are you going to tell your vendor how urgent your need is for cognitive search?

Editor's Note: This article has been updated to . 

fa-solid fa-hand-paper Learn how you can join our contributor community.