2014-01-December-Interpreters-G20.jpgAmong the many tools organizations use to evaluate enterprise content management (ECM) solutions lies Gartner's research, specifically its Magic Quadrant (MQ), which scores ECM solutions and solution providers in specified categories. But before you turn to the MQ when seeking your next ECM solution, take the following factors into consideration.

Gartner provides an excellent picture of each ECM vendor’s general health, impact on the ECM market, ability to fulfill its own objectives and overall breadth of offerings. But its analysis is market- and vendor-focused, rather than implementation- and customer-focused. 

Using the 2013 report as an example, the category names and descriptions make the distinction between market focus and implementation focus clear: Completeness of Vision: Market Understanding, Marketing Strategy, Offering (Product) Strategy, Vertical/Industry Strategy, Innovation, Geographic Strategy; Ability to Execute: Product/Service, Overall Viability, Market Responsiveness/Record, Customer Experience, Operations.

None of its categories in Completeness of Vision or Execution directly address success of implementations, resource efficiency (in terms of cost, time, required skills, people), risk of implementation failure or customer satisfaction. While some of the categories are relevant, the focus is typically superficial and on inputs, rather than an empirical assessment of post-implementation outputs and impact. For example, the most relevant category, “Customer Experience,” evaluates the kinds of programs the vendor offers, rather than their impact. The biggest vendors can easily win this category, even when actual customer experience is poor.

For Gartner, the vendors’ vision, strategy and ability to execute are measured primarily against the requirements of the vendors themselves and the very general “ECM market.” They are not measured against the customers in general, customers in your vertical industry (e.g. financial services, manufacturing, government) or your organization in particular.  

Specifically: Gartner’s evaluation does not address the particular requirements of these segments or of your organization, which differ from the general market. It is also high level in the sense that it is not empirically based on assessing actual deployments of the software or on close evaluation of the post-implementation success and customer satisfaction of each vendor’s products within actual implementations (beyond high level survey questions).

While the 2014 ECM MQ report took some steps to rectify this with the inclusion of  evaluations for specific contexts -- small and midsize companies, European companies, those in financial services -- the point remains. Analyzing software outside the context of an implementation leaves out a big part of the picture.

Finally, the Gartner evaluation does not directly address the total cost and risk of each vendor for customers -- the “downside” that is necessary in evaluating your options.

How to Evaluate High Scoring Vendors

When you look at the high scoring ECM vendors in the Gartner Magic Quadrant, several vendors score high on Gartner’s criteria, but do far less well when:

  • Evaluated from a customer-centric perspective. Many of the big vendors fail or are severely challenged here.
  • The products are evaluated in actual production. Many products look good on paper (on a slide deck, in a demo), but can’t do the job for your situation when it counts -- when fully loaded with documents, users, integrated systems and active tasks in the flow. Many vendors require resource-intensive supplementation and workarounds whose necessity only become evident deep in the first phase of the implementation.
  • The specific functional and other requirements of customers are taken into account. Note, however, that in most instances the products fail not because they are “bad” but rather because they are extended into applications for which they were not designed. Many vendors don’t pull themselves out of such opportunities where they aren’t a good fit for the customer, and the Gartner analysis doesn’t discourage such dangerous overextension.
  • The resource requirements (particularly cost) and risk are included in the assessment. The biggest, most mature and advanced ECM vendors do relatively poorly on this, but generalists like Microsoft also do poorly when requiring supplementation that is not productized by a third party.

How to Evaluate Lower Scoring Vendors

Some of the ECM vendors that score lower on the MQ score high in customer-centric and empirical criteria.

Although many vendors score poorly across the board and should be rejected outright, some don’t make it into the MQ net, but warrant further consideration for your organization and opportunity. They may score low in MQ because of their relatively narrow specialization within ECM -- which eliminates them on the basis of "Completeness of Vision" -- and score high in the customer-centric and empirical criteria.

How to Evaluate Vendors Not Included in the MQ

Your ECM solution selection process should often include several non-MQ vendors, particularly in the early phases when you are developing your “long list” of candidates. These providers are not included in the MQ for several reasons, but typically just don’t fit the ECM suite mold. For example, they might offer document management capabilities, but not broad ECM (Box). Some focus on specific vertical markets (e.g. life sciences) or broader verticals (e.g. manufacturing). Some of these vendors may be highly viable candidates for you -- or at least included in your assessment out of due diligence.

Title image by Downing Street (Flickr) via a CC BY-NC-ND 2.0 license