This is a excerpt from the new Corporate Buyers’ Guide to LLMs, published this week by GAI Insights. In producing the report, the analyst firm interviewed consultants and vendors, as well as chief technology officers, chief digital officers, chief innovation officers, and other company leaders at 40 prominent organizations.
• • •
A GenAI customer care project is now helping save $4 million per year for the popular car insurance firm, Jerry. This GenAI project is already implemented and saving money.
Jerry is a company that aims to help consumers manage everything related to car ownership in one app, including comparing car insurance quotes, finding repair shops and quotes, and reaping savings through safe driving.
Since its inception in 2017, Jerry has experienced rapid growth, serving five million customers. The company is fueled by $110 million in funding, received a $450 million valuation in 2021, and just earned a spot on Forbes’ 2023 Best Startup Employers list. While the company is obviously thrilled about its success, its customer service model began experiencing stress that threatened to disenfranchise some customers.
At the inaugural Generative AI World Conference 2023 in Boston, Jerry’s COO, John Spottiswood, shared how using LLMs, the underpinnings of text-to-text GenAI, helped Jerry improve customer response times and save the company.
The Problem: Scaling Customer Service for Rapid Growth
With a focus on a fully digital and real-time experience, Jerry receives more than 200,000 messages a month from more than 100,000 unique users. The high volume meant the company struggled to respond to customer queries within 24 hours, let alone minutes or seconds.
The Solution: Deploying an LLM-driven Chatbot
Jerry decided to leverage LLMs to create a chatbot named “Kelly Bota.” The chatbot was designed to handle routine queries, freeing human agents to tackle more complex issues. The technology stack involved using OpenAI’s GPT-4 for complex queries and OpenAI’s GPT-3.5 for initial sorting, integrated via APIs. Messages are captured from chat and SMS through Twilio and stored on Jerry’s servers. The chatbot’s responses are then sent to Front, a third-party customer service app, and finally to the users.
Execution: A Multi-agent System
Jerry developed a multi-agent system where each virtual agent specializes in different subjects. For instance, the “App Screens” agent identifies if a user’s issue can be resolved through the mobile app, and directs them accordingly. The “Routing” agent decides which agents should handle a particular user request. The system also includes “Payments,” “Policy,” and “Opt-out” agents, among others.
Jerry employed a JavaScript library called Handlebars to insert data from its database into the chatbot’s responses. It also developed a robust testing framework that allows it to evaluate the chatbot’s performance both pre-launch and post-launch.
Jerry started the effort in January 2023 with just two junior engineers, a project manager, and CTO involvement. It has some additional talent since, but the team remains small.
The chatbot was launched in May 2023, and by June 2023, 100 percent of inbound messages were responded to within 24 hours, up from just 54 percent in April 2023
Impact: $4 Million Per Year Savings
The results have been spectacular. The chatbot was launched in May 2023, and by June 2023, 100 percent of inbound messages were responded to within 24 hours, up from just 54 percent in April 2023. Today, 96 percent of messages are responded to within 30 seconds. The number of queries requiring human intervention has dropped from 100 percent to just 11 percent.
Financially, the chatbot has already achieved a 55 percent ROI as of August 2023, with an expected 400 percent ROI and $4 million in annualized savings by year-end.
Keys to Success
Jerry’s success can be attributed to several factors: a focus on prompt engineering, rapid iteration, investment in testing, and version control. Jerry executives started small, adding resources only when they saw tangible results.
Jerry plans to introduce more specialized agents and is looking to test new LLMs. It also aims to launch a voice agent and implement a lightweight customer feedback system by the end of 2023.
This case study is an excerpt from The Corporate Buyers’ Guide to LLMs, a report published by GAI Insights in November 2023. InnoLead members receive a 50 percent discount on the report purchase price using the code below. (This code will only be visible if you are signed in as a member.)
InnoLead members: enter the discount code InnoLead50 to receive 50 percent off the report purchase price.
