How to Measure Chatbot Success: 8 Ways to Do It
Updated: Aug 15, 2019
Chatbots, companies such as KLM, Univé, Mediamarkt and Booking.com already make use of it. Other brands and organizations are keeping a close eye on developments. Despite a
growing number of organizations that use it, marketers struggle with the way in which the success of chatbots can be reported.
The development of chatbots
In the United States and Asia it has been commonplace for some time. Chatbots that can help consumers in their search for the answer to a pressing question or that one product.
Conversational commerce is very much the trend. Chatsbots are pieces of software that use the interface of chat apps and automatically develop themselves on the basis of machine learning.
The use has also gained momentum in Europe because Facebook Messenger and Slack have been opened to companies. Earlier this was already possible for the less popular chat applications WeChat, Telegram and Kik.
Of course there are also challenges. Fully automatic is in an era where consumer information like personal contact and relevance are not always optimal. There are also more and more
initiatives that combine a 'stupid' chatbot with human interventions.
In addition to machine learning, Natural Language Processing (NLP) also plays an important role in this.
The bot learns to learn from these human interventions and develops itself further and further. The use of chatbots can therefore no longer be stopped. Marketers have to start optimizing them for the maximum effect. However, it must be clear when a chatbot is a success.
When is a chatbot a success?
Marketers have become quite spoiled in recent years. In addition to reach and engagement, issues such as customer satisfaction and incremental turnover compared to control groups are also being reported. The complete (sales) funnel can be monitored, further optimized and a report on the ROI be produced where necessary.
In the field of messaging apps and in particular chatbots, standard metrics are not yet available. For AdvertisingAge, this is a reason to take a closer look at this and marketers who start chatbots to put a tick against eight possible measurement points for chatbots.
Every marketer will want to know how many people use their chatbot, unique and in absolute numbers. Without further context, these are actually meaningless numbers.
Where do you initially put it against?
Previously there was already the Canadian chat application Kik that came with definitions of active users . In order to monitor the development in use (and with it the progress), it is also advisable to report the number of active users per week, per month and per year.
Definition of a session & average session time (range & engagement):
When is there a chat session? Is a question and answer sufficient? Or does a weekly push notification with the latest offers count as a session? Even if no response from the receiver follows? Does someone count who comes back an hour later after asking a question as one or two sessions? This will have to be clearly defined beforehand. The definitions will depend on the industry, just as there may be varying objectives. Then report on the number
of sessions and average session time.
Number of sessions per user (engagement):
Another tricky one. Is a low or high frequency good? It depends entirely on the service you offer. In the case of customer service, for example, you hope that a bot can help a consumer as quickly and optimally as possible within a session. If that consumer then has to start several new sessions to solve problems, that is not desirable.
Number of interactions per user (engagement):
In line with the number of sessions per user, it is also interesting to look at the number of interactions within a session.
Each question-answer must bring someone closer to his goal (solving a need or question). It may be that a chatbot intuitively reaches the goal via a number of detours. Machine learning is therefore extremely important to reduce the number of interactions.
Confusion triggers (engagement):
Anyone who has ever had to deal with a chatbot may have received a response that there is no solution at hand. Reactions like 'I do not know' or 'I cannot help you' must be monitored. Marketers must learn from these loose ends and where necessary take them away in the future.
Click-through rate (engagement):
We also know these from other channels. In many cases, chatbots will refer consumers to a specific page. The clicks / offered clicks ratio is an excellent tool for monitoring. Note that click-through rate by itself is not sufficient, additional web analysis based on that click will have to be included.
Lead generation (conversion):
A subscription for the newsletter, scheduling an appointment, a download. Just a few examples of possibilities that a bot can offer to make it easier for the consumer.
In addition to improving customer satisfaction, chatbots will develop more and more as a means of selling something. Attribution will initially be a challenge, but was this not the case with other channels?
To think about
I discussed eight measurement points where marketers can measure the first success of a chatbot. But is this all-encompassing? There are still a number of elements that influence chatbots that are difficult to measure in many cases.
• Branding: With a growing number of consumers using chatbots, it is also possible to measure NPS and purchase intent.
• Research: Another channel, that means more data! How do we deal with this data to further optimize our business operations?
• Content curation: Consumer input can also be used for future campaigns. Think of frequently asked questions.
• Opt-out: As with other channels, consumers will also get enough of chatbots. What does marketeers say when a chatbot is blocked? Did it not help the consumer in his
or her question? Or were the weekly messages a bit too commercial?
• Cannibalization: The emergence of chatbots will affect other channels and services. Is the pressure on the customer contact centre relieved and the costs go down?
Will chatbots lead to increased push marketing (push notifications or e-mail) or replace pull marketing (search) partially?