Customer Story

Betterment Raises Chatbot BSAT, Containment, and Automated Resolution Rate with QA

Watch the Webinar
Industry
Financial Tech
Use Case
Chatbot QA
Company Size
500+ employees

Increase in CSAT

30%

Decrease in agent ramp time

120%

Increase in monthly coaching sessions

300%

Chatbots often serve as the first point of interaction between brands and customers. When their performance is subpar, it can negatively impact customer satisfaction. Natalie Langdale, Chatbot Strategy Manager at Betterment, understands the importance of getting chatbot performance right. In this article, we share Betterment's approach to Chatbot QA and the techniques used to boost chatbot accuracy, efficiency, and overall experience through MaestroQA.

Challenge: High Volume, Compliance, and Limited Visibility in Reporting

Betterment’s rapid growth presented several challenges when it came to maintaining chatbot performance and customer satisfaction:

  • Managing high ticket volumes: With a growing customer base, the number of inquiries increased significantly. Betterment needed their chatbot to handle basic queries while allowing human agents to focus on complex issues.
  • Ensuring compliance in a regulated industry: As a financial services company, Betterment had to routinely review and check that their chatbot didn’t provide inaccurate or misleading advice—a serious concern in such a regulated industry.
  • Limited visibility in reporting: While Betterment’s chatbot platform, Ada, provided high-level metrics such as containment rate and automated resolution rate, the data was not granular enough to identify the root causes of issues or knowledge gaps in chatbot responses.
  • High-level metrics that don’t show the full picture: The chatbot tool’s basic stats failed to give a complete understanding of where the bot was falling short, making it difficult to refine performance and help maintain consistent, accurate responses.

We established a really rigorous QA process for our bots. A key part of getting approval from our legal, risk, and compliance teams was proving that we had a systematic way to review and improve our generative bot’s outputs, which was entirely enabled via our Maestro integration."

Natalie Langdale

Chatbot Strategy Manager

Solution: A QA Program That Delivers Granular Insights

Betterment implemented a robust Chatbot QA program with MaestroQA, allowing them to address both the performance and compliance challenges while gaining deeper insights into chatbot interactions.

  1. Building a Comprehensive QA Rubric Betterment developed a tailored QA rubric to evaluate their chatbot on four critical areas:
  • Accuracy: Ensuring that the chatbot understands the user’s inquiry and responds with the correct information.
  • Completeness: Verifying that every inquiry is fully addressed.
  • Compliance: Ensuring that the bot adheres to strict financial regulations and does not provide misleading or incorrect advice.
  • Clarity: Maintaining a conversational tone that aligns with Betterment’s brand while being clear and concise.
  1. Using MaestroQA to Dig Deeper: While Ada provided general metrics, MaestroQA allowed Betterment to go beyond surface-level data. The platform enabled detailed analysis of chatbot conversations, helping identify specific gaps in knowledge and opportunities for improvement. By reviewing individual interactions, the team was able to improve chatbot responses in real time.
  2. Regulatory Compliance with RAG (Retrieval-Augmented Generation): To avoid compliance risks, Betterment implemented RAG technology. This model supports the bot’s ability to cite verified sources, reducing the likelihood of generating incorrect or inappropriate responses. The integration of MaestroQA further strengthened this by providing detailed feedback on the bot’s performance.

Results: Greater Performance and Compliance, Enhanced Visibility

With a focus on both chatbot performance and compliance, Betterment achieved substantial improvements:

  • Improved Visibility into Performance: The combination of MaestroQA and Ada allowed Betterment to move beyond high-level metrics, providing detailed insights that enabled continuous improvement of chatbot interactions.
  • Better Compliance Management: RAG technology and regular checks helped maintain the chatbot’s adherence to legal standards, reducing compliance risks and increasing trust in chatbot responses.
  • Accurate and Relevant Responses: Detailed analysis of conversations through MaestroQA allowed the team to fine-tune responses and close knowledge gaps, significantly improving the quality of chatbot interactions.

Since implementing Chatbot QA, Betterment has seen the following improvements to their chatbot performance:

  • BSAT: +5%
  • Containment Rate: +2%
  • Automated Resolution Rate: +2%

Impact: Holistic Chatbot QA for Consistent Improvement

Betterment’s Chatbot QA program supports their bots in meeting rigorous standards of accuracy, efficiency, and compliance. By integrating both bot and human CSAT data, Betterment can monitor the entire customer journey and help maintain a seamless, high-quality experience across all touchpoints.

As Betterment continues to grow, their focus on granular insights and continuous QA ensures that their chatbots evolve in line with customer expectations, providing reliable and compliant support at scale.

Ready to refine your chatbot performance? Natalie Langdale shares detailed insights on how Betterment uses MaestroQA to boost chatbot accuracy, efficiency, and compliance. Watch the full webinar and discover how to take your chatbot strategy to the next level.

Webinar

Perfecting Chatbot Performance

Chatbots are often the first point of contact with your customers, and their performance can make or break the customer experience—don’t leave their performance to chance. Join Natalie Langdale, the driving force behind Betterment’s chatbot strategy, in a webinar that’s all about maximizing your chatbot’s effectiveness with MaestroQA.

Webinar

Quality & Risk Operational Excellence in CX

Discover how TaskRabbit transformed its approach to risk management, leveraging data-driven insights to enhance safety protocols, optimize training, and drive continuous improvement. Hear firsthand from TaskRabbit's team as they share their challenges and strategies. Learn how MaestroQA's tools can revolutionize your risk management practices!

Webinar

Next-Level Agent Performance

Discover the unique blend of empathy and efficiency that characterizes Getaround's modern coaching model. Learn how they use data-driven coaching conversations to strengthen the core relationship between team leads and frontline agents, reinforcing their core values and enabling them to adapt quickly to dynamic customer needs.

Customer Story

Heading

Watch the Webinar
Industry
This is some text inside of a div block.
Use Case
This is some text inside of a div block.
Company Size
This is some text inside of a div block.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

Webinar

Quality & Risk Operational Excellence in CX

Discover how TaskRabbit transformed its approach to risk management, leveraging data-driven insights to enhance safety protocols, optimize training, and drive continuous improvement. Hear firsthand from TaskRabbit's team as they share their challenges and strategies. Learn how MaestroQA's tools can revolutionize your risk management practices!