Content Management System (CMS) News, Reviews, Events and Analysis.
 
 
 

CMS Selection - Death to the Features Matrix

By Seth Gottlieb  | Published Mar 31, 2009

Researching and selecting a web content management system is often an arduous process. On the outset, few organizations realize what they are getting into. And most team reps and decision makers have other daily responsibilities that compete for their attention. It's not surprising that simple looking decision making tools suddenly become attractive.

Once the requirements collection and product evaluation phases are complete, the most difficult part of a CMS selection process starts. In this phase the group must take all the information that was gathered and meaningfully use it to make an accurate decision. With project fatigue setting in, this is where things often go wrong.

From Requirements to Decisions

This translation phase is where people often get a bit wild with spreadsheets and scoring, all in the hopes that math will heroically make a complicated and confusing (and, lets face it, subjective) decision obvious and irrefutable.

The process looks something like this. There are a bunch of selection criteria. People rate the products on each criterion. People weight the criteria. A spreadsheet is used to perform some multiplication and addition and voila! out comes some a very quantitative looking assessment.

Numeric Simplicity Saves the Day?

Nothing looks more convincing than a score where one option has more points than another.

But, users don’t necessarily want to use a system just because it has the highest cumulative, weighted score. They want to use a system that helps them efficiently get their jobs done while introducing the fewest number of annoyances.

If the measurement of accuracy is the overall satisfaction with the solution, this method is extremely faulty.

Matrix Scoring is Rarely Accurate

There are several reasons why the matrix scoring method fails to accurately select the right solution.

First, the rating and weighting wind up being very subjective and arbitrary. Veterans of this approach know this to be true when the they remember the feeling of not knowing what to put down or wanting to change their score when they see another product or have more coffee.

Second, the final score hides information that is important to the users. A typical example is where a user finds a critical (to him or her) feature totally unusable but that is overshadowed by excellent ratings in a majority of less important features.

Usually you can’t correct this with weightings — especially if there are lots of selection criteria. You can’t discuss trade-offs and compromises if you are just working with total scores.

Lastly, criteria tend to be of unequal granularity. How can a broad criteria like “usability” be compared with something as specific as “SSL on the login page?”

cms-feature-doubt-matrix-01.jpg
A Bogus Selection Matrix

Doubt Provokes Discussion, Casts Light on Key Concerns

I take a different approach to the decision making process. Instead of forcing the selection committee into making numerical ratings, I ask them to list their doubts with each solution.

Examples of doubts are:

  • a concern that the feature would not support a specific task
  • unnecessary complexity or awkward behavior in doing a specific task
  • an unsatisfactory explanation by the supplier about how a feature worked
  • doubt about the vendor’s stability or ability to support the customer
  • a potential technical incompatibility with the legacy infrastructure

Each of these doubts are investigated as whether they are valid — that is, if it was a misunderstanding or oversight, if there is a suitable work-around, or if there is a reasonable compromise.

Through a number of facilitated sessions, we work through comparing the relative weaknesses of the competing solutions and determining what is tolerable.

Follow-up demos and calls with the vendors are scheduled and executed. Ultimately, the solution with the fewest legitimate and significant concerns wins.

Facilitating these sessions is not as easy as simply reporting matrix scores but I think that it is good that people put some real intellectual energy into making such an important and complex choice.

Best = Set of Least Painful Compromises

At first glance, this system seems designed for selecting the lesser of evils and to some extent that is true — there is no such thing as a perfect content management system. There will always be compromises involved in the final decision (I should note here that there is an option of selecting nothing if no solution is good enough). But is this approach really any worse than a numerical system that decides a 5 out of 1000 score is better than 3 out of a 1000. I think not.

Three Key Benefits

A Clear Focus

The first benefit of the doubt technique is that it keeps the focus on things that have real impact on the users of the CMS — forcing the group to think through the implications of specific aspects of the solution. This is better than having a people register their concern in the sparse format of a low numerical score and then just move on.

Deeper Understanding

The second benefit is that selection committee members interactively learn more about their needs and more about the software features as they watch demos and work with prototypes. As a result, their selection criteria become more sophisticated over time and potentially critical information is allowed to enter the decision making process at any time.

Post Decision Posture

The third key benefit is that after the product is selected, the selection committee can all clearly verbalize the reason behind the decision. If there is a complaint about the implemented solution, the  selection committee can refer back to previous analysis sessions, express that the problem was  identified as a concern and then explain the plan to lessen the impact.

Overall, by making the process more discussion and investigation oriented, one encourages additional richness in the discovery process and the team exits with the realistic perspective that they are embarking on an investment with a tool that while imperfect, is at least deficient in ways that the group has consciously decided they can work with, or around.

 

About the Author

Seth Gottlieb is the founder and principal of Content Here, an analyst firm and consultancy specializing in content technologies. With 15 years of experience in software and professional services, Seth has helped businesses large and small improve the efficiency and effectiveness of their content management and publishing processes. He has been a regular contributor to CMS Watch and is the author of Open Source Web Content Management in Java among other critically acclaimed reports.

Speak with Seth at the Gilbane Conference in San Fracisco (June 2, 2009). He'll be presenting a workshop on how to choose a web content management system.

 
Read More About:
, , , ,
 
Was this article useful?
 

4 Reader Comments

1 | Ivan Mironchuk — March 31, 2009 3:44 PM

I think that if you are working with a business analyst or consultant that understands how functional requirements should be written, then the functional requirements matrix can be very helpful. Vague requirements like "user friendliness" shouldn't even be included on a functional requirements matrix.

We take an approach where we work with our clients to determine discrete functional requirements i.e. you may say you want an easier way to search for things, but what you really mean is that you are looking for Full Text Search across the following X document types. You aren't just looking to integrate your CMS with another business system, you are looking to use SOAP services to pull assets from X location along with metadata defined in an XML format.

If you break out these functional requirements into categories then you can have the client weight the categories along with the individual functional requirements. This helps skew scoring results based on areas of functionality that really speak to process pain points.

The scoring of these functional requirements can be used to narrow a very wide field of vendor candidates down to a few finalists. These finalist vendors should then be brought in to do live demonstrations of their product, so the all important usability factors can be seen first hand.

In conjunction with the functional requirements it should be determined what are the key measures of success in implementing a new system and if the functional requirements speak directly to the key success criteria.

This process has proven to work very well with our customers.

2 | Joe Bachana — April 1, 2009 9:07 AM

Seth, this is awesome! (I'm depressed that Ivan got to this post before me though).

Recently I have also gotten down on the long list of functional requirements as the primary way to numerically score the vendors's products. However, I find that the mere act of documenting those requirements early on (pre-implementation) gets people thinking about what they need to achieve with their Web Content Management System.

The functional requirement matrix also becomes a useful guide (or precursor) to the implementation team that will come in after product selection, and it also is helpful to see the gap between what the selected product has out-of-box vs what has to be built/customized/integrated w/ a 3rd party tool.

The act of scoring doesn't really take a lot of time, so I think it is useful, although I emphatically agree that peoples' scoring will change based on whether they've seen any product demos at any point prior to the scoring work. A bit like keeping the jury cloistered during a major trial to shelter them from the news media, I suppose.

One thing we have found pretty helpful for screening the long list of vendors and products is having our customers look at the bigger requirements -- those YES/NO things that if the vendor doesn't have it, they're taken off of consideration. The list can be anywhere from 5-15 items and range from technology platform (if a customer is .NET or LAMP or J2EE and has those kinds of resources, why bring anything else in?), vendor competency (you cite this above), and major feature requirements.

For my part, our job (yours and mine and the other consultants) is to help the customers winnow down the choices to a manageable list of possibilities that will work within the customers' environment -- then, let the customer work on the qualitative decisionmaking. That will be governed by demonstration review, meetings with the vendor to see how they conduct business, learning more about their business practices through the CMS community and through other end users, and so forth.

Thanks for the great article, Seth. Keep 'em coming!

3 | Seth Gottlieb — April 1, 2009 10:31 AM

Thanks Ivan and Joe for your comments. I probably should have clarified that I didn't mean that buyers should not collect requirements and prioritize them. Requirements like you describe are good for filtering but not good for final decision making - at least not the approach generating cumulative scores.

The doubt approach is only used after the number of options has been filtered down to four or less and a decision needs to be made. However, I do think that companies go astray when requirements analysis gets so granular that they get over 100 requirements.

Creating enough descriptive text around each requirement so that everyone has a shared understanding of the meaning can take forever; and nobody actually bothers reading a document that long anyway. I take the approach of writing scenarios for primary uses of the system. Each scenario is a narrative of what the user needs to do. At the bottom of each scenario, I have a bulleted list of requirements that were demonstrated in the scenario.

For non-functional requirements, like the system has to run on Red Hat and MySQL, a tersely worded requirement is fine. Here I am looking for powerful market filters, which I call "leading requirements," that can quickly eliminate large segments of the market.

Many of the leading requirements are non-functional and are a simple yes/no. I go into more detail about my approach for requirements gathering and short listing in this article Leading Requirements.

4 | Eric Brown — April 25, 2009 9:51 AM

Excellent take on features matrices. With so many options out there, everyone competing at a rate that is staggering, and many CMS options going almost micro-niche (directing at the publishing industry, the news industry, etc.) the scores that a feature or features may get may have no bearing on what my intended use is. Therefore high scoring features may mean nothing in the end.

For example Plone may have some of the most likely used features for me, but if I look at a features scoring matrix I will probably see that Plone installation can be tougher than others and runs on Zope + Python not PHP and MySQL (although Plone talks to nearly any database). Without real investigation it may score lower than others overall based on this and I would miss out due to lack of quality research.

I think what it comes down to is assessing needs and doing good investigative research to see what options best meet those needs in a real sense.

Thanks, Seth, for the post.

Leave a Response

  Remember me?

Related Web CMS Articles

 

From our Job Board  View all jobs | feed Jobs RSS feed | Post a job right now

 

Featured Events  View all events | feed Events RSS feed | Add your event

STAY UP TO DATE
Subscribe to our RSS feed...
SUBSCRIBE TO OUR RSS FEED

Just Commented On




Resources