measuring Chatbot success

Some of your customers prefer to work with chatbots as they connect with you, according to Inbenta’s Chatbot Consumer and Business Survey. Their data shows that 50 percent of consumers prefer chatbots when shopping online, rather than calling support. And 72 percent of consumers say that chatbots hold the answer to frustration-free customer service.

While others reports tell a different story. For example, a recent report from PointSource, a Globant company, found that 80 percent of retail customers aren’t comfortable with chatbot assistance when resolving problems post purchase.

The lesson? It’s a nascent space, but so long as you’re deploying chatbots in the enterprise, you can measure their success to ensure they’re actually helping customers or your employees, whichever experience you'd like to improve. We caught up with a few experts to discuss key actionable metrics for your chatbot programs.

Related Article: Are Your Chatbot Initiatives Underperforming? Chatbot Silos Are Likely The Reason

Start By Defining KPIs

Draw out your KPIs and the ways to measure them, both quantitatively and qualitatively, said Ranga Srinivasan, president, CTO and co-founder of Ameex Technologies. “Everybody is learning the best way to formulate metrics to evaluate the bot performance, as is the case with any new technology. With bots we do not have a reference to compare it with, but some key traditional metrics still very much hold good and apply here, too,” Srinivasan said. 

He suggests a number of quantitative and qualitative measurements, including the following. 

  • Total sale value, direct to customer
  • Conversion rate
  • Customer support savings
  • Increase in Net Promoter Score
  • Cost of operations and maintenance
  • Cost per acquisiton
  • Number of active users 
  • Number of bot sessions initiated 
  • Average daily number of sessions/user
  • Average daily number of chats handled by bot
  • Number of new users using bots daily, weekly, monthly
  • Intent-based analytics
  • Which intent has the most exits?
  • Lift in sentiment engagement
  • Overall customer retention rate.

Human Interaction Vs Chatbot Interaction

Jordi Torras, CEO of Inbenta, posits that chatbot efficiency has to be compared with the live chat human agent. “Most probably, when measuring these KPIs, we will find a variety of results. In some cases chatbots will do better than humans, in some other cases it will be exactly the opposite,” Torras said.

Torass suggested the following relevant KPIs when comparing: 

  • Conversion rate — To which extent a chat agent transforms conversations online into new business. "That could apply to the entire online sales process, or mere appointment setting services. In any case we will compare the percentage that, in average, our human chat agents are getting to what chatbots are obtaining."
  • Self-service rate — To which extent a chatbot is able to solve conversations by not creating a case that has to be solved by a second-tier call center determines this rate. 
  • Satisfaction rate — Every conversation should be given the option to be rated by the customer. A 5-star scoring system would be ideal, after every chat conversation, but more complex surveys could also be deployed. Ideally, customers should be offered some sort of compensation for their ratings, like a future discount, a coupon, or some other incentive, according to Torras.
  • Confusion triggers — Some bots will get confused when trying to understand user questions — and eventually some humans too, according to Torras. "Having an efficient way to see when the conversation entered an 'awkward' moment is not easy, and might require selecting samples of past conversations to estimate this rate," he said.
  • Artificial Intelligence (AI) and Machine Learning (ML) rates — Probably more important than measuring these KPIs from a static perspective, measure them from a dynamic point of view. "That means," Torras said, "having a consistent measure of how a chatbot ML improves KPIs over time, and equally important is the ability to measure how individual human chat agents learn to be more efficient including what is the time for new agents to ramp up into the job."

Measure Task Success Rates

Sometime chatbots are deployed in-house. And just like customers, employee experience matters, too. Scot Whigham, director of global IT service and support at InterContinental Hotels Group (IHG), has helped his staff deploy chatbots internally as technical support for the hotel as well as its corporate colleagues. Task success is a major category for chatbot metrics, according to Whigham. IHG measures percentages of tasks completed vs. tasks that had to be escalated to live human agents and efficiencies within that process. They’ll measure error rates within the task itself. “There are two main ways that we look at task success. Did they execute on the task to the satisfaction of the customer calling in or did we have to escalate the next stage which was a human being? Were there any errors throughout the tasks that went through? Did it flow smoothly and was the intent understood?” Whigham said.

The ultimate goal is to ensure the chatbots understand the customer's language and know enough about the products and services in order to understand intent.

Consider Scalability and User Retention

Chatbots can’t simply support one user or one module at a time. Therefore scalability is key, according to Christian Pedersen, chief product officer for SAP S/4 HANA Cloud. “Companies need to consider the retention of users measured by repeated use, successful conversion and impact on revenue when looking to deploy chatbots,” Pedersen said. 

Analyze ‘Unproductive Interactions’

Determine root causes of unproductive interactions, Pedersen added. Natural Language Processing (NLP) is evolving, and chatbots are making inroads recognizing user intention behind requests even with unintentional input typos. However, if users cannot be helped by the chatbot, then the interaction requires human specialist intervention. “Companies that measure the rate of intervention from chatbots to humans can more effectively train chatbots to manage an autonomous interaction from inquiry to resolution,” Pedersen said. 

Related Article: 8 Brands Innovating with Chatbots

Set Benchmarks, Note Sentiment

Organizations should consider the ideal succession of exchanges between user and chatbot, Pedersen said, and the optimal time frame of resolution. It can be used as a foundational metric for chatbot performance. “The opportunity to learn about user behavior through integrating chatbots and analytics is massive,” he added. “Sentiment analysis should be used to understand how users feel about the conversation progression relative to content relevance.” 

User Adoption, Retention

IHG’s Whigham said his team pays close attention to retention and adoption. They want to know how many users in their population are interacting with chatbots for the first time. How many of them come back after the initial visit? “Retention and adoption: those are two metrics that are very important to us,” Whigham said. 

Related Article: Does Your Company Need a Chatbot?

Track Engagement Metrics

Whigham said his team is also interested in discovering details on the frequency that users take advantage of a chatbot platform, and the intensity and depth of the interactions. “How deep did we go into your issues, how deep into the tasks did we go and then how much time you’re spending doing basic stuff,” Whigham said. Intensity, he says, is how complex of an issue is it that they're trying to address for their customer and then how many layers does it go?”

Engagement metrics can yield what Whigham called “objective metrics” but can also help determine the quality of the interactions between the platforms and understanding intent.