taking the plane out for a test spin
Promises on paper don't always live up to real life. Here are some tips for running a successful web CMS pilot PHOTO: Inge Wallumrød

The wrong CMS can act as an anchor, weighing down your brand’s online presence — which is the last thing you need in a market that rewards companies for riding the waves created by whichever new and exciting channel has emerged. 

Sure, tried and tested methods of choosing a CMS are out there, and it usually is pretty clear when a change is needed. But no matter how much you discuss the issue, you never quite know if you’ve made the right call until you and your team have your hands on the software.

And that’s where the pilot comes in. By spending some time with software, before purchase and before a full-scale roll out, your company will have time to kick the bumpers and compare how the software is in a real setting as opposed to on paper. 

Why Pilot?

A software pilot allows organizations to roll out software on a small scale in real-world conditions to test the effectiveness of the software in question. A pilot reduces the risk of an unsuccessful organization-wide implementation later, enabling the organization’s staff to determine if this software is the appropriate solution.

Unlike a trial period, the purpose of the pilot is primarily to prove viability, rather than deliver an agreed upon outcome.

Once a pilot is considered successful, the organization can roll out the environment to the entire end user community. 

To better understand the benefits of a pilot scheme, we asked a few experts to weigh in.

The Question

What are the benefits of a CMS or DXP pilot, and what best practices do you recommend for brands considering a pilot?

Tony White, Founder of Ars Logica

Tony White
Tony White
Tony White founded Ars Logica, a vendor-neutral analyst firm, six years ago. He and his team help companies evaluate their web content management, customer experience management and ecommerce requirements by helping them select appropriate software. Tweet to @ArsLogica

Probably the clearest benefit of pilots is validation of the business case for a new technology platform. Since companies often don’t have a clear understanding of what’s actually in a new DX platform — either from feature-functional or technical/architectural perspectives — a pilot provides some hands-on time for marketers and IT resources to apply a platform in a real-world setting.  

For gaining a basic understanding of how a system might actually be used, there is no comparison between a vendor demo and a pilot — the pilot (also frequently called a proof-of-concept) is infinitely more valuable. 

It’s worth mentioning here that we often recommend two proofs-of-concept when a customer is having a difficult time deciding between two vendor finalists. Since vendors will often comp a customer for a proof-of-concept at the time of purchase, the customer ends up really paying for only one of them — that cost pays for itself many times over in the long run.

One major pitfall to avoid is having only a technical vetting of a platform during a pilot. It’s not uncommon for marketers to think they understand the feature-functionality of a DX platform and to have the technical team use a pilot just to “kick the tires” technically. This is dangerous, because a platform might actually be a good fit with an IT organization and yet not be able to address the branding and marketing goals of the organization as a whole.

A pilot should include a decent chunk of information architectural planning and taxonomy work, and it should very clearly define the scope of what is to be accomplished. The scope definitions/limits are critically important. With scope discipline, keeping the pilot cost effective isn’t too difficult. 

Once a customer has a well-defined scope, it can simply present the use cases (both technical and marketing) to a vendor for an estimate. This is the time a customer should ask a vendor about having the pilot cost comped back once a purchase agreement has been signed. It’s a little trickier with an agency [feel free to probe me here if you like], but at least the vendor time/resources used during the pilot could be comped back.

It’s also a fairly common practice for a vendor to provide serious prospects a remote instance (“sandbox”) of the platform that it can use for proof-of-concept purposes. Oftentimes, the remote instance will be sufficient to complete the PoC.

We’ve had customers recently who’ve struggled with PoC/pilot issues for over 3 years after purchasing a technology platform. One of those customers recently abandoned their chosen platform before actually getting it fully implemented, at a cost of more than $1 million. They went to a simpler platform, because they found that it met their real needs — not the inflated “goals” they defined in their PoC. The real cause of the failure was a lack of effective communication between marketing and IT.

Tony Byrne, Founder of Real Story Group

Tony Byrne
Tony Byrne
Tony Byrne has spent the last 16 years of his career evaluating the strengths and weaknesses of digital workplace and marketing technologies for the benefit of companies making early-stage software adoption decisions. Tweet to @TonyByrne

The prime benefit of a pilot is to “try before you buy.” The key is for all the stakeholders (editors, marketers, designers, developers) to get hands on with the finalist solutions. With our clients, we have found that, more often than not, the CMS vendor who came in second after the demo round ends up winning the Proof of Concept (PoC) round: evidence that the proof is in the doing, not talking.

The best approach to a PoC is to do your diligence first and make it very clear that you are down to two finalists — so the vendor is invested in a real opportunity and not a fishing exercise.  

Our standard PoC template for WCM platforms calls for a week-long pilot, though we have led some that are shorter and occasionally longer: it really depends on the complexity of the environment and the level of customer investment in the ultimate solution. A media firm for example (for whom content is their product), will want to do deeper diligence than a manufacturing firm looking to power their public website.

Deane Barker, Partner and Chief Strategy Officer at Blend Interactive

Deane Barker
Deane Barker
As Co-Founder, Partner and CSO of Blend Interactive, Deane Barker spends his time helping brands manage their web and content struggles. Barker also wrote the book, "Web Content Management: Systems, Features and Best Practices." Tweet to @Gadgetopia

The best reason to run a pilot is to get a handle on what you're actually going to do. So many dreams are sold by vendors that few customers can ever hope to implement. By running a pilot, you can get a feel what you might actually do, rather than what are just myths and legends that a vendor is using to sell the product.

As for best practices, keep staffing in mind. Just because something works on a small scale doesn't mean you're going to be able to staff it on a large scale. These systems don't run themselves. They're a "force multiplier," clearly, but you will still have to apply staffing, and if you don't have the staff now, you're probably not going to magically have them in the future. Don't buy into something that requires you to rely on humans you don't already have.