1. Interaction volume
2. Average duration of the session
3. Total number of users
4. Sessions per user
5. Interactions per user
6. Consultation resolution rate
7. Transfer rate to humans
8. Response time
9. Drop-out rate
10. Follow-up or user satisfaction
11. Engagement
12. Retention rate
13. Conversion rate
14.Customer satisfaction rate (CSAT)
In this post we are going to address the metrics that you should take into account to ensure the success of your users’ interaction with your AI chatbot. And to make it easier for you to understand, we are going to give you examples.
It is important to keep in mind that if you want to achieve great results with your AI chatbot, you will need to constantly improve it. To know where and what to improve, you need to track and monitor, on a recurring basis, the main metrics and, therefore, you can have the necessary information to identify opportunities for improvement and optimisation in general and in particular terms.
These are the essential metrics to measure the success of your users’ interaction with your chatbot that you should take into account. Let’s get started!
1.Interaction volumne
Number of sessions initiated, number of messages sent and received and frequency of use.
Measures how often users interact with the chatbot.
Example: A customer service chatbot records that it has had 10,000 interactions in a month. Each interaction can be broken down into sessions, messages sent and received, and responses provided. This metric helps to assess the traffic and popularity of the chatbot in specific periods (peak demand, most active times of the day).
This graph shows the frequency of chatbot usage over time, helping to identify days of peak demand. The blue line shows the number of sessions initiated daily, while the green and red lines represent messages sent and received respectively. The peaks indicate days of higher interaction, allowing us to observe usage patterns, such as higher activity on certain days.
2. Average duration of the session
Average time users spend in a chatbot conversation session.
It evaluates the efficiency of responses and the clarity of the conversation flow.
Example: The average session length is 5 minutes, suggesting that users are spending enough time to resolve their queries. Too little time could indicate a lack of usefulness, too much time could indicate a frustrating or low-quality experience.
This graph indicates how quickly and efficiently the chatbot satisfies queries. Long durations may reflect users’ difficulties in finding clear answers; while short durations may indicate efficiency, but also a lack of depth of information, it shows the frequency of sessions in intervals of duration, which allows us to observe whether sessions are mostly short or long. Most sessions are concentrated between 2 and 5 minutes, suggesting a reasonable time to resolve queries without causing frustration to users. This analysis helps to understand the efficiency of the chatbot, as excessively long durations could indicate problems of understanding or clarity in the answers.
3.Total number of users
Total number of unique users who have interacted with the chatbot during a given period.
Measures the reach and popularity of the chatbot among users.
Example: After one month, the chatbot has 20,000 unique users. This growth is stable and positive, indicating that the chatbot is being used by new users on a regular basis. This metric helps to identify the expansion of the user base and to adjust the chatbot’s promotional strategies or functionality to maximise its reach.
This graph helps to assess the adoption and popularity of the chatbot. A steady growth in the number of users indicates a good level of acceptance and visibility. A slowdown or decline may suggest that better promotion of the chatbot or adjustments to its functionality are needed. It shows the cumulative growth over a month. It highlights peaks on specific days (Day 10, Day 20 and Day 30), which may coincide with campaigns or special events that increased the number of users. This type of graph allows you to gauge the adoption and popularity of the chatbot and continued growth indicates a good level of acceptance.
4. Sessions per user
Average number of sessions initiated by each user in a given period.
Measures the recurrence of user interaction with the chatbot.
Example: In a month, the average number of sessions per user is 3. This indicates that users regularly return to the chatbot, suggesting an interest or need to use it for ongoing assistance. An increase in the average may reflect the chatbot’s effectiveness in retaining users.
This graph helps to understand the frequency of returning users, indicating retention and satisfaction. A steady increase suggests an engaged user base, while a decrease may signal the need for improvements to the chatbot experience. In this example, we see a steady increase in the average number of sessions per user over the year, suggesting an engaged and satisfied user base. An increase in this metric indicates that users find value in the chatbot and return regularly to interact with it.
5. Interactions per user
Average number of messages sent and received by each user during a session.
Measures the amount of exchange needed to satisfy a query.
Example: If the average is 10 interactions per session, it indicates a reasonable flow of conversation where users are finding satisfactory answers. If the number of interactions is excessively high, it may indicate that the chatbot needs to improve the clarity of its responses.
This metric reflects the efficiency and flow of the conversation. A high number of interactions might indicate that users are asking for clarification or more information, while a balanced number indicates a smooth and clear conversation. This bar chart illustrates the average number of interactions per user at monthly intervals. In this example, we see a gradual increase in the number of interactions per user over the year. This metric helps to understand the efficiency and flow of the conversation.
6. Consultation resolution rate
Measures how many user questions or problems were successfully resolved by the chatbot without the need for human intervention.
Example: Out of 5,000 queries received, the chatbot resolved 85% without human intervention. This indicates that the chatbot can handle a large volume of queries autonomously. If a user asks for help on the status of an order, the chatbot should be able to provide this information successfully and without additional help.
Indicates the effectiveness of the chatbot in answering queries, with a high resolution rate representing good performance. The blue bar represents queries resolved by the chatbot (85% or 4,250 of queries), while the orange bar shows queries transferred to human agents (15% or 750 queries). This high percentage of resolution without human intervention indicates good effectiveness in the chatbot’s ability to handle queries autonomously.
Note: This metric can be complemented by the human handover rate metric, the percentage of conversations that handover to human agents.
7. Transfer rate to humans
Measures how many conversations are transferred to human agents. A low rate indicates that the chatbot is capable of resolving queries on its own.
Example: Of all interactions, 20% were transferred to a human agent. This metric indicates how many queries are beyond the reach of the chatbot and need human intervention. An increase in this metric may signal the need to train the chatbot on specific topics or expand its knowledge base.
A high percentage indicate that the chatbot needs adjustments in its programming or further training on specific topics. The graph shows that 20% of queries were transferred to a human agent, while 80% were solved directly by the chatbot. This type of visualisation makes it possible to assess the chatbot’s ability to resolve queries autonomously and to identify opportunities for improvement in its programming or training.
Note: This metric can be complemented by the query resolution rate metric, the number of interactions that are resolved without human intervention.
8. Response time
The time it takes for the chatbot to respond to users’ questions. A quick response contributes to a better experience.
Measures how quickly the chatbot responds to users; an average of 1-2 seconds can indicate optimal performance.
Example: The chatbot responds to queries in an average of 2 seconds. This is especially important in customer service situations where fast response times improve user experience and retention in the conversation. A longer response time could indicate performance issues or congestion in the system.
Histogram showing chatbot response time in seconds. Most response times are around 2 seconds, indicated by the red line, suggesting optimal performance. This type of analysis allows us to see if the chatbot maintains fast response times, which are crucial for a good user experience, especially in customer service.
9. Drop-out rate
The percentage of users who leave a conversation without the chatbot resolving their queries.
Indicates whether the chatbot is failing to provide useful or engaging answers.
Example: 30% of users leave the conversation before receiving a satisfactory answer. This percentage allows you to identify at which specific points in the conversation users drop out (such as when asking about pricing or specific features). This may suggest that the chatbot needs to improve on certain responses.
This graph helps to identify at which step of the conversation users drop out, indicating areas for improvement in the dialogue flow. Each point shows the number of users at each stage of the conversation, indicating where the interaction decreases. It helps to obtain at which stages users drop out, pointing to areas for improving the flow of the dialogue.
10. Follow-up or user satisfaction
Some chatbots measure user sentiment during the interaction, using natural language analysis to detect emotions (positive, negative or neutral) and adjust responses or measure satisfaction.
This is measured with natural language analysis and classifies sentiment into categories that reflect the user’s experience.
Example: The chatbot uses natural language analysis to detect whether responses contain positive (‘thank you’, ‘perfect’) or negative (‘frustrated’, ‘lousy’) words. If 75% of interactions have a positive sentiment, the chatbot is meeting its goal of providing a good experience.
A feedback system can also measure this metric in the form of quick post-interaction satisfaction surveys.
This bar chart illustrates how a chatbot can use natural language analysis to classify the sentiment of user interactions into positive, neutral and negative categories. In this example, 75% of interactions have a positive sentiment, indicating that the chatbot is meeting its goal of providing a good user experience.
11. Engagement
Evaluates how users interact with the chatbot in terms of messages per session and length of conversations.
Example: Users interact with the chatbot an average of 5 times per session and each session lasts about 3 minutes. This level of engagement indicates that users find value in the chatbot and do not leave the conversation quickly.
This graph allows you to see whether users stay in the conversation and how they interact throughout the session. It illustrates how users interact with the chatbot. The horizontal axis shows the average number of messages per session, while the vertical axis represents the duration of the session in minutes. This type of graph allows you to visualise whether users stay in the conversation and how they interact throughout the session. In this example, users interact with the chatbot an average of 5 times per session and each session lasts around 3 minutes, indicating a good level of engagement.
12. Retention rate
How many users interact with the chatbot again after their first conversation. This is a measure of the chatbot’s long-term usefulness and attractiveness.
Retention rate indicates whether users find value in the chatbot and want to return.
Example: 40% of users who interacted with the chatbot interact again in the following month. This metric indicates success in creating a repeatable and positive experience, motivating users to return.
This line graph shows the percentage of users who return to interact with the chatbot each month. In this example, 40% of users who interacted with the chatbot in January interacted again in February, and so on. The gradual decline in retention rate over months can help identify areas for improvement to maintain user interest and satisfaction over the long term.
13. Conversion rate
In commercial environments, it measures how many interactions led to a desired conversion, such as filling out a contact form.
Useful for measuring effectiveness in guiding the user to perform commercial actions, such as obtaining leads.
Example: Out of every 100 users who interact with the chatbot, 15 convert into leads. This rate is a key metric in commercial chatbots and is used to measure the effectiveness of the chatbot in guiding the user to a specific action.
This conversion funnel illustrates the percentage of users who go through different stages to complete a desired action. In this example, out of every 100 users who interact with the chatbot, 50 continue to interact and finally, 15 complete the contact action. This metric is crucial to measure the effectiveness of the chatbot in guiding users towards specific business actions, such as converting them into leads.
14. Customer satisfaction rate (CSAT)
Some chatbots request a rating or direct feedback from the user after the interaction to assess the quality of the service.
At the end of the interaction, direct feedback shows overall satisfaction and provides insights for improvement based on the user’s perception.
Example: At the end of the conversation, the chatbot asks the user to rate their experience from 1 to 5 stars or leave a comment. With an average of 4.5 stars, the chatbot can measure satisfaction directly and receive suggestions for improvement.
In conclusion, these metrics combined with attribute dimensions and transactional, registration, support and customer service categories, by which you can filter, group and segment, will allow you to adjust and optimise the experience of your AI chatbot, to improve the achievement of KPIs such as satisfaction, query resolution rate, user retention, number of transactions, etc…, in short, to improve and refine all the KPIs you consider necessary in order to obtain maximum performance from your chatbot.
We hope that with this post we have been able to tell you in a clear way how to manage the behaviour of an AI chatbot, in relation to the metrics that are obtained related to the interaction with users, all this in order to obtain maximum effectiveness within the different digital channels that you are using to attract, retain, convert, build loyalty and optimise based on your strategy for growth and success in the medium and long term.
Optimising chatbots does not have a universal formula, as each case is unique. Therefore, the best option is to have experts in the field who can help you take your chatbot to the next level in an effective and personalised way.
If you are looking for a solution for your business we will be happy to help. You can book a free consultation with us to get an expert opinion and offer you solutions tailored to your needs. Let’s talk!
Sources and bibliography: