Feature
The chatbot industry is expected to hit $1.3 billion by 2025. Further, 37% of Artificial Intelligence (AI) startup founders expect chatbots and virtual agents to be the top consumer applications for AI over the next five years, according to numbers released by AI Multiple this month.
The numbers are promising, but naturally chatbots come with their caveats. This still remains a nascent space. Customer experience leaders need to closely examine whether or not chatbots best serve their customer experience management programs.
“Chatbots aren’t at maximum maturity yet, so strongly defining their purpose, scope and function on your digital properties plays a huge role in their success,” said Ranga Srinivasan, president, CTO and co-founder of Ameex Technologies. “Defining these should be an organization's first priority before starting down a path of implementing chatbots.”
And, as you’re deploying chatbots in the enterprise, you can measure their success to ensure they’re actually helping customers or your employees, whichever experience you'd like to improve. We caught up with a few experts to discuss key actionable metrics for your chatbot programs.
Understand Business Challenges, Necessary Improvements
Before thinking about metrics or defining chatbot program success, Srinivasan said organizations should understand the business challenges they're trying to solve or what they’re trying to improve by using a chatbot. Define the scope the bot is going to play in addressing these challenges or improvements, Srinivasan added.
“Chatbots work best to accomplish a few things," he said. “Serving quicker answers to customers’ questions, resolving complaints, feedback or issues, directing users to more information when they search for something, engaging customers by serving content, prompting them to check out, up-selling, cross-selling and so on. The first few examples solve customer service and satisfaction business goals, and the latter serve customer engagement, retention and conversion.”
Related Article: Chatbots Provide the Personal Touch. Here's Why That's a Danger
Define Scope Your Bot Plays in Addressing Challenges and Improvements
Understanding what type of business goals you’re trying to solve leads nicely to the next step, Srinivasan said. A bot is just one tool in your arsenal to solve business challenges you identify. Once you identify the challenges, understand where and how the bot can help solve these goals, according to Srinivasan.
He suggested some common ways bot help:
- They have a knowledge base of questions and answers to directly answer your customers’ questions, reducing time for customers to find an answer.
- They redirect your customers to content so they can quickly help themselves, improving the customer experience.
- They send notifications at key moments in the customer journey to accomplish your business goals (sign up for a newsletter, check out their cart, view a recommended product etc), increasing conversion rates for your business
- Increase user engagement via surveys, questionnaires or interactive experiences, improving brand affiliation and customer engagement/satisfaction
- Automate after sales services and support
“Usually, it’s after understanding the business challenges that your bot is going to address and defining its scope that you define the KPIs which you’re going to measure against to validate whether the bot you’ve implemented actually serves its purpose,” Srinivasan said.
Define Your KPIs
Draw out your KPIs and the ways to measure them, both quantitatively and qualitatively, Srinivasan said. “Everybody is learning the best way to formulate metrics to evaluate the bot performance, as is the case with any new technology. With bots we do not have a reference to compare it with, but some key traditional metrics still very much hold good and apply here, too,” Srinivasan said.
He suggests a number of quantitative and qualitative measurements, including the following:
- Total sale value, direct to customer
- Conversion rate
- Customer support savings
- Increase in Net Promoter Score
- Cost of operations and maintenance
- Cost per acquisition
- Number of active users
- Number of bot sessions initiated
- Average daily number of sessions/user
- Average daily number of chats handled by bot
- Number of new users using bots daily, weekly, monthly
- Intent-based analytics
- Which intent has the most exits?
- Lift in sentiment engagement
- Overall customer retention rate
Related Article: How to Take Your Chatbots to the Next Level
Human Interaction Versus Chatbot Interaction
Jordi Torras, CEO of Inbenta, posits that chatbot efficiency has to be compared with the live chat human agent. “Most probably, when measuring these KPIs, we will find a variety of results. In some cases chatbots will do better than humans, in some other cases it will be exactly the opposite,” Torras said.
Torass suggested the following relevant KPIs when comparing:
- Conversion rate: To which extent a chat agent transforms conversations online into new business. "That could apply to the entire online sales process, or mere appointment setting services. In any case we will compare the percentage that, in average, our human chat agents are getting to what chatbots are obtaining."
- Self-service rate: To which extent a chatbot is able to solve conversations by not creating a case that has to be solved by a second-tier call center determines this rate.
- Satisfaction rate: Every conversation should be given the option to be rated by the customer. A 5-star scoring system would be ideal, after every chat conversation, but more complex surveys could also be deployed. Ideally, customers should be offered some sort of compensation for their ratings, like a future discount, a coupon, or some other incentive, according to Torras.
- Confusion triggers: Some bots will get confused when trying to understand user questions — and eventually some humans too, according to Torras. "Having an efficient way to see when the conversation entered an 'awkward' moment is not easy, and might require selecting samples of past conversations to estimate this rate," he said.
- AI and Machine Learning (ML) rates: Probably more important than measuring these KPIs from a static perspective, measure them from a dynamic point of view. "That means," Torras said, "having a consistent measure of how a chatbot ML improves KPIs over time, and equally important is the ability to measure how individual human chat agents learn to be more efficient including what is the time for new agents to ramp up into the job."
Measure Task Success Rates
Sometime chatbots are deployed in-house. And just like with customers, employee experience matters. Scot Whigham, managing director at Function-AI. told CMSWire that in his previous role he helped his staff deploy chatbots internally as technical support for a hotel as well as its corporate colleagues.
Task success is a major category for chatbot metrics, according to Whigham. IHG measures percentages of tasks completed vs. tasks that had to be escalated to live human agents and efficiencies within that process. They’ll measure error rates within the task itself. “There are two main ways that we look at task success. Did they execute on the task to the satisfaction of the customer calling in or did we have to escalate the next stage which was a human being? Were there any errors throughout the tasks that went through? Did it flow smoothly and was the intent understood?” Whigham said.
The ultimate goal is to ensure the chatbots understand the customer's language and know enough about the products and services in order to understand intent.
Related Article: Keep These Three Things in Mind When Introducing Chatbots to Customer Service
Complexity of Issues and User Demographics
Rakesh Jayaprakash, product manager for ManageEngine, said that though chatbot algorithms have become advanced in the past few years and are more accustomed to human language since their initial days, people still tend to have mixed feelings about their experience with chatbots.
Learning Opportunities
Webinar
Jun
6
The Future of Social Media: Emerging Trends and How to Stay Ahead of the Curve
The impact of emerging technologies like AI on social media
Webinar
Jun
7
The Building Blocks of Personalization
This webinar will explore how personalization can create powerful ecommerce shopping experience.
Webinar
Jun
8
Attracting—and Keeping—Your High-Net-Worth Investors
Join PwC to dissect their recent survey findings on a specific customer subset — high-net-worth investors.
Webinar
Jun
13
The Future of Contact Centers with AI at the Helm
Learn how to balance human intuition and interaction with the potential uses of AI to create delightful and lasting CX
Webinar
Jun
13
Learn How to Deliver Dynamic Video Experiences at Scale
Learn how to convert existing assets into video experiences in near real-time.
Webinar
Jun
20
3 Steps to Unlock Your Customer Data and Drive Revenue
Discover how Twilio Segment’s #1 Customer Data Platform helps implement data-driven strategies.
“Besides looking at the obvious metrics such as the percentage reduction in cases logged in your help desk by virtue of a chatbot program, organizations should consider two important factors while measuring success: complexity of issues and user demographics,” Jayaprakash said.
One of the main reasons there is varied opinion about chatbots amongst organizations concerns is the complexity of their customers’ problems. Customers of a bank or other financial institution often reach out for technical help regarding billing issues that might require some investigation.
“Chatbots may have to evolve further in order to perform such complex operations,” Jayaprakash said. “Also, organizations may have limitations on the amount of information they can expose to the chatbot for analysis. On the other hand, ecommerce companies that deal with questions of lesser complexity, such as 'Where is my order?' or 'Is my item eligible for return?' may find their support load reduced by 50 percent or more.”
Consider Scalability and User Retention
Chatbots can’t simply support one user or one module at a time. Therefore scalability is key, according to Christian Pedersen, chief product officer at IFS.“Companies need to consider the retention of users measured by repeated use, successful conversion and impact on revenue when looking to deploy chatbots,” Pedersen said.
Determine root causes of unproductive interactions, Pedersen added. Natural Language Processing (NLP) is evolving, and chatbots are making inroads recognizing user intention behind requests even with unintentional input typos. However, if users cannot be helped by the chatbot, then the interaction requires human specialist intervention. “Companies that measure the rate of intervention from chatbots to humans can more effectively train chatbots to manage an autonomous interaction from inquiry to resolution,” Pedersen said.
Related Article: 7 Ways to Use Chatbots Effectively in Customer Experience
Set Benchmarks and Note Sentiment
Organizations should consider the ideal succession of exchanges between user and chatbot, Pedersen said, and the optimal time frame of resolution. It can be used as a foundational metric for chatbot performance. “The opportunity to learn about user behavior through integrating chatbots and analytics is massive,” he added. “Sentiment analysis should be used to understand how users feel about the conversation progression relative to content relevance.”
User Adoption and Retention
Whigham said his team pays close attention to retention and adoption. They want to know how many users in their population are interacting with chatbots for the first time. How many of them come back after the initial visit? “Retention and adoption: those are two metrics that are very important to us,” Whigham said.
Related Article: Conversational Marketing: How to go Beyond Chatbots
Track Engagement Metrics
Whigham said his team is also interested in discovering details on the frequency that users take advantage of a chatbot platform, and the intensity and depth of the interactions. “How deep did we go into your issues, how deep into the tasks did we go and then how much time you’re spending doing basic stuff,” Whigham said. Intensity, he says, is how complex of an issue is it that they're trying to address for their customer and then how many layers does it go?”
Engagement metrics can yield what Whigham called “objective metrics” but can also help determine the quality of the interactions between the platforms and understanding intent.
What’s Next for Chatbots?
In 2021, customer service representatives will rely more on automation technology to support additional demands and overwhelmed staff, according to Steven Petruk, president of the Global Outsourcing Division for CGS.
This will be most evident in such industries as retail, where many businesses have exhausted resources after the 2020 holiday boom, even as many large retailers increased tech support. Petruk noted that FedEx committed to adding more technology for the holiday crunch on top of the pandemic ecommerce boom, and Amazon invested in new solutions for its warehouse facilities.
“In addition to interactive AI chatbots, we can expect to see automation aid customers through a “see what I see” (SWIS) technology via augmented reality (AR) for in-home consumer support as well as for field services support in the telecommunications and hospitality industries,” Petruk said. “After a year of social distancing, our customer service survey confirmed that for technical support, more human contact is desired. And, our chatbot research found 71% of consumers would be less likely to use a brand if it didn’t have human agents available. As we enter 2021, companies need to integrate the ability to access a human for more complex or unsolved issues to ensure the best customer experience.”
