Benchmarking Your Chatbot's User Engagement
How to measure success in a completely new environment?
Chatbots are incredibly new.
Before hopping on the exciting bandwagon and starting to plan your chatbot, it is worth asking yourself this simple question: how will I measure success?
Measuring your chatbot's user engagement is one way to go. Although not perfect, measuring engagement should give you a better idea of how well your chatbot is performing.
This is all well and good, but before we start measuring anything at all, we need a way to benchmark what we find.
Unfortunately, as they are so new, there is a lack of data on chatbot performance, user engagement, retention, click-through rates, and the like.
In this article, we will suggest how to measure your chatbot's user engagement in an actionable way as well as explain why you shouldn't listen to some of the negative hype.
Comparing user engagement with known channels
Although it is hard to find user engagement data for chatbots (as of 2016), there is one thing you can do: compare with known channels.
The idea is to benchmark your chatbot's performance versus the performance of the channel you are currently using for the same task. Let me explain with an example.
Say you currently own an e-commerce website selling handmade woolly hats. At the moment, you sell approximately 50 hats per day (crushing it by the way, congrats).
After each sale, you send two emails to your new customer.
The first is sent right after the purchase to thank the person for their custom.
The second is sent a week later. The goal is to get feedback on your product, by asking your customer to rate it on a scale from 1 to 10.
Having read that mobile messaging is the best alternative to email marketing, you think it would be a good idea to replace this process with a chatbot.
Existing channel: email
New channel: chatbot
To benchmark your fancy new chatbot, you would simply take current results from the existing channel and compare it to the result you get with your new channel.
Simples!
Not all that scientific!
Now, I know what you're thinking...
"This is not all that accurate."
You are right.
This thing is, though, at the moment this is the best you can do. As you will see below, comparing your chatbot's engagement metrics against other chatbots does not make sense.
There are only a handful of chatbots in the wild that have a big enough audience to generate useful metrics.
Even if you did have access to the data, how could you compare a news chatbot to the pretend feedback chatbot we describe above?
Not possible.
If user retention is a big question for you and a deciding factor of whether or not you are going to have a chatbot built, this is the way to decide: how does (or would) your chatbot perform versus your current method (be it email, phone, letters, etc.)?
Run a small test, compare your results, and make an educated decision based on a benchmark.
Don't believe the (negative) hype
It is hard to ignore the negative hype around chatbot engagement and user retention. Some have shared data showing huge drop offs in users after only a couple of messages.
Although I am absolutely not contesting these numbers, it's important to remember not all bots are alike.
Sure, some bots have seen up to 65% drop off in usage after message two, but what was the purpose of the bot? Was it providing value to users?
It is hard to believe a real value-providing bot would lose that many users in such a short period of time.
The value-adding bots we have built show an amazingly low churn rate between 3.14% and 0.29%... after message 6!
Churn rate, or the velocity at which users are unsubscribing from your chatbot, can be a great indicator of user engagement. Typically, if a user is engaged they will not churn. It is the ultimate metric.
Making your own benchmark and reporting
As you can see, there is still a lot to learn about chatbots, engagement, churn, and other business metrics. The next step for you as an aspiring chatbot owner is to define your own benchmark and reporting.
To benchmark your chatbot user's engagement, do the following.
1. Define the process you are going to replace with a chatbot in your business. This can be any sort of outreach, automation, interaction, FAQ, customer support, etc.
2. Dig through the data you currently have on this process. Using email? Connect to your email provider and find the usual suspects (open rates, click-through rates, etc.).
3. If you don't have a chatbot yet, create a SMART goal for user engagement vs. current state. At which point would you consider your chatbot successful over your current efforts?
4. If you do have a chatbot already running, dig through your data and find the corresponding numbers.
5. Compare. It's as easy as plotting two lines on an Excel chart.
6. Pick the best one (duh)!
