Even with the best of intentions, project timelines can be lead astray.
During the many years helping clients create new intranets or installing a new search application, two processes always cause the project timeline to go off course: The first is in forecasting the time it takes to undergo a proof of concept exercise, and the second is migrating content to the new application.
David Hobbs covers migration planning at great length, so in this column I want to consider the management of a proof of concept (PoC) exercise, with special reference to enterprise search.
Elements of a Proof of Concept
A proof of concept must include a number of attributes:
- It has to be realistic (is this how users will use it every day?)
- It has to be scalable (it works with 100 items, but what about 100,000?)
- It has to be extensible (it works for the IT Department, but what about R&D?)
- It has to be able to be evaluated (because the proof has to be unambiguous)
Designing a PoC to meet these four attributes is not easy.
Scalability in Search
Two performance indicators for search are recall and precision.
Both are a function of relevance and determine what items should be returned in response to a specific query. This is the point when you'll discover views on relevance differ wildly. It is worth reading a history of test collection experiments to get a sense of the experimental difficulties.
You may need a number of different test collections at this point, using search logs to determine which queries to use to assess the outcomes. I do not recommend using an arbitrary chunk of content from the previous year for this.
Running a Proof of Concept: Parallel or Sequential?
The next issue is whether you will run a PoC to narrow down your choice between a few applications, or as due diligence on the preferred vendor.
How you answer depends on some very practical considerations.
Running PoCs in parallel takes a considerable amount of staff time and scheduling. Scheduling requires coordination with the vendor, who will have staff trained specifically to undertake PoC tests, but perhaps not in the country you want to run the tests in.
That will lead to discussions about both scheduling and travel costs. While it may be tempting, do not set a date for PoC tests in the RFP. Doing so may deter potentially attractive vendors from bidding because you look like a client who has no sense of reality.
The other option is to run the PoCs sequentially. The danger here is if one of the early PoCs runs overtime, it will impact the schedule for subsequent PoCs.
Learning Opportunities
Webinar
Mar
29
The CX Commute vs. the CX Joy Ride
Is your CX traffic holding up your customer's journey?
Webinar
Mar
29
Is Your Microsoft 365 Data Recoverable? How to Survive the IT Hotseat When Data is Lost
See why MIcrosoft recommends third party backup
Webinar
Mar
30
An Optimized Customer Contact Strategy Combines Transparency and Intelligence: The State of Outbound Communications
Learn from the Neustar-commissioned Forrester Consulting study on challenges and solutions for contact centers
Webinar
Apr
13
Accelerating Content Velocity Across the Digital Supply Chain
Learn to integrate DAM with upstream tools to improve workflow efficiency.
Conference
May
10
CMSWire CONNECT Customer Experience Conference - Austin 2023
Don't miss the top customer experience and digital experience conference of the year — live in Austin, Texas May 10-12.
Conference
May
10
Reworked CONNECT Employee Experience & Digital Workplace Conference Austin 2023
North America's best employee experience and digital workplace conference of the year — live in Austin May 10-12, 2023.
Another common problem is when the first PoC brings to light some fundamental issues the client needs to address (security management is usually on the list). That vendor will often ask to run the PoC again as their competitors will benefit from the discoveries and remedies.
Coming up with a scoring model to compare search applications is difficult — believe me. Since (hopefully) you will have scored the initial proposal, you will then not only have to score the PoC but also decide what weight to give the PoC score against the proposal evaluation score.
The other option is only to run PoC tests on the preferred application for the initial evaluation. While in theory this will reduce the amount of work involved, if the PoC fails you will have to start the process all over again with the next in line.
Will the Solution Be Out of the Box or Customized?
Another important issue with search is the extent to which you expect the vendor to customize the application and/or the user interface ahead of the PoC.
Vendors are (naturally) not very keen on doing this unless there is a payment for the development effort. However, out of the box search applications are very basic, as any user of SharePoint will appreciate. Running a PoC on a base configuration is a waste of time when both you and the vendor know that the customization of the application is the reason for considering it.
Still Want to Do a Proof of Concept?
I hope this brief summary gave you a sense of the challenges PoC assessments create. My experience suggests many of the questions a search procurement team has about an application can be addressed in meetings with current clients of the vendor. I'm talking serious three-hour meetings around a defined agenda, not a quick chat over Skype.
Vendors may push back on this, for fear that commercial information about discounts will be exchanged.
If nothing else, remember this: PoCs cannot remedy a failure to specify requirements clearly in the RFP. The RFP is the cornerstone of a search procurement and it has to be grounded in a search strategy. The more effort you put into the RFP, the less the requirement for PoCs — and the greater the chance of staying on schedule and staying sane.
Learn how you can join our contributor community.