To benchmark your fancy new chatbot, you would simply take current results from the existing channel and compare it to the result you get with your new channel.
Not all that scientific!
Now, I know what you're thinking...
"This is not all that accurate."
You are right.
This thing is, though, at the moment this is the best you can do. As you will see below, comparing your chatbot's engagement metrics against other chatbots does not make sense.
There are only a handful of chatbots in the wild that have a big enough audience to generate useful metrics.
Even if you did have access to the data, how could you compare a news chatbot to the pretend feedback chatbot we describe above?
If user retention is a big question for you and a deciding factor of whether or not you are going to have a chatbot built, this is the way to decide: how does (or would) your chatbot perform versus your current method (be it email, phone, letters, etc.)?
Run a small test, compare your results, and make an educated decision based on a benchmark.
Don't believe the (negative) hype
It is hard to ignore the negative hype around chatbot engagement and user retention. Some have shared data showing huge drop offs in users after only a couple of messages.
Although I am absolutely not contesting these numbers, it's important to remember not all bots are alike.
It is hard to believe a real value-providing bot would lose that many users in such a short period of time.
The value-adding bots we have built show an amazingly low churn rate between 3.14% and 0.29%... after message 6!
Churn rate, or the velocity at which users are unsubscribing from your chatbot, can be a great indicator of user engagement. Typically, if a user is engaged they will not churn. It is the ultimate metric.
Making your own benchmark and reporting
As you can see, there is still a lot to learn about chatbots, engagement, churn, and other business metrics. The next step for you as an aspiring chatbot owner is to define your own benchmark and reporting.
To benchmark your chatbot user's engagement, do the following.
1. Define the process you are going to replace with a chatbot in your business. This can be any sort of outreach, automation, interaction, FAQ, customer support, etc.
2. Dig through the data you currently have on this process. Using email? Connect to your email provider and find the usual suspects (open rates, click-through rates, etc.).
3. If you don't have a chatbot yet, create a SMART goal for user engagement vs. current state. At which point would you consider your chatbot successful over your current efforts?
4. If you do have a chatbot already running, dig through your data and find the corresponding numbers.
5. Compare. It's as easy as plotting two lines on an Excel chart.