Tracking the Right Chatbot KPI
Although parallels can be made between chatbot technology and other means of communication (emails, live chat, phone calls, etc.), a chatbot remains a unique beast.
As such, businesses must figure out which chatbot KPI they need to track to measure success.
With this article, I want to help you find the right KPI to track by listing some of the main ones we monitor for our clients. The 'right' KPI for your exact business case may not be in here, but my hope is the list should give you a starting point and some inspiration to move things forward.
A chatbot KPI that works for you
Before digging in, I have to give the usual 'make this your own' speech.
The KPIs below are not the panacea. Your business is unique. As much as I'd like to give you The One Chatbot KPI to Rule Them All, that is just not within my power.
Please, do not just pick one (or more) below randomly. Make sure you study your business goals and what you are trying to achieve with your bot. If you are at a loss, I recently wrote an article on benchmarking chatbot KPIs which could be useful to you.
1. Activation Rate
I tend to cringe a little bit when chatbot owners talk about 'number of users' as their leading KPI.
As most people in the marketing industry know, acquiring users is not the end goal. Although an important step, acquiring users is merely just making them aware of your product - not turning them into successful users.
Therefore, a much better chatbot KPI would be Activation Rate.
Read about AARRR if you need to refresh your memory.
The activation rate is the number of users that go beyond the initial interaction (acquisition) and on to perform one more task that matter to your chatbot's goal.
Here's an example using the theScore Facebook Messenger chatbot. Their goal is to keep me up to date on my favourite football team.
They have a Get Started button. Once pressed, this is what I get.
Right now, they have acquired me as a chatbot user. But, really, how engaged am I so far? I have not done anything. I have not indicated any interest in any team or what the chatbot has to offer.
Their Activation Metric could be 'Team followed'. As soon as I actually pick a team to follow, I display a real interest in their service. It may not be the best metric for their business, but it is far better than 'Acquisition'.
Think beyond acquisition and towards activation. What is one action you think all your successful chatbot users need to take? Measure that KPI.
2. Interactions per User
This is an easy one: measuring the number of messages sent and received by a user.
Although I have a love/hate relationship with this metric, I do think it is one you must keep a pulse of at all time.
Why do I love this metric? Because it is a good sign of engagement. It is a base metric for many other metrics.
Why do I hate this metric? Because, in a way, it is a vanity metric which by itself doesn't say much. An engagement metric does not indicate the quality of the engagement.
As an example, here is my interaction with KiviHealth, a chatbot aimed at facilitation patient to doctor interactions.
Though as a unique user I have had 8 interactions, this exchange is completely fruitless.
Think of the Interactions per User metric as the Page Views metric of a website. It is at the root of many calculations but in itself doesn't give much insight into the quality of your product.
Measure it, don't swear by it.
3. Voluntary User Engagement
More interesting than the simple Interactions per User, the Voluntary User Engagement metric shows the number of time a user has engaged with the chatbot without being prompted to do so.
Interesting things happen when you couple this metric with the retention metric (see below).
Keep a pulse on the number of times your users come back and voluntarily message your chatbot. Initiating unprompted interactions is a great way to measure engagement and actual interest in what you have to offer.
4. Retention Rate
Of course, we could not have a chatbot KPI article without sneaking in the timeless (pun intended) Retention Rate.
The retention rate is usually represented in the percentage of users who come back to your chatbot within a specific period of time.
Much has been said about retention rate in chatbots. Some say as high as 40% of chatbot users never make it through day 1. Ouch.
We personally have never worked on chatbots with such low retention rate. In a recent article, I wrote about increasing retention and how one of our clients enjoyed a mere 0.29% churn rate on day 10 of their chatbot campaign.
Clearly, this is a metric you should keep an eye on. Measure your retention on day 1, 7, 15, 30, and 90.
5. Goal Completion Rate
When it comes to actionable insights, this metric is quite clearly up there.
The Goal Completion Rate is exactly what it sounds like. Out of all the users who message your chatbot, how many actually reach the goal you initially set the chatbot for?
If your chatbot's goal is to deliver a quote for one of your services to its users, how many of all the people who message it actually end up with a quote in hand?
This metric will directly impact your business' bottom line, your chatbot optimisation path, and even whether or not your chatbot is successful overall.
6. Fall Back Rates
There are three types of fall back we use in all the chatbots we build. Each of them serves its own purpose and thus should be measured separately.
6.a. Confusion
Sometimes, chatbots get confused. This most likely happens when a user sends an unexpected input.
Our answer to this is simply to let the users know we don't understand what they are saying. A simple 'Sorry, I did not get that. Can you rephrase?' usually does the trick.
Measuring the Confusion Rate (number of time the chatbot had to fall back divided by total number of messages sent) will give you an indication of the work you might have to do in training your bot's artificial intelligence.
6.b. Recalibration
Sometimes, users get confused. When a user sends a series of messages that are out of the scope of the chatbot's understanding, it sometimes helps to recalibrate the conversation.
To do this, we tend to use the confusion fallbacks twice, then fall back to a recalibration on the third unrecognised user input.
A simple 'Looks like we're lost. As a reminder, I can do X, Y, and Z for you. For instance, you can ask me XYZ'.
By showing the user exactly how to use the chatbot, we try to recenter the conversation towards a) what the chatbot can understand/do, and b) the chatbot's goal.
6.c. Human
The fallback of all fallbacks: humans.
Sometimes, as intelligent as your chatbot is, a human is needed to swoop in and take control.
Keep an eye on the number of times your chatbot actually had to revert back to humans for help. Quick sidenote: make sure this is an actual fall back and not a chatbot goal. If you release a sales chatbot, eventually talking to a human is a great thing!
This concludes our series of chatbot KPIs to track. To reiterate, make sure you adapt these to your business goals before you do anything else!
