A voice agent that answers customer calls and recommends the right product

Client Industry

Retail & e-commerce · online store operations

Key Challenge

Slow, low-concurrency phone support for orders & product help

Core Tech Stack

LiveKit WebRTC · Twilio · STT/TTS · RAG · Python

Deployment Model

Concurrent isolated agent workers on LiveKit Cloud

SoftmaxAI has developed a production-quality conversational voice agent which customers can access by dialing a regular telephone number in order to check the status of their orders and receive personalized product recommendations — this is a true telephony system, not a pre-programmed IVR. Twilio receives the incoming call over the public switched telephone network (PSTN) and connects it to a LiveKit WebRTC agent worker, the interaction being powered in real time by speech-to-text, language understanding, retrieval-augmented grounding, and function calling to enable a natural conversation. Each caller is assigned a separate worker within an isolated environment so that the conversations remain independent and reliable even when there is a high volume of calls underway — and a swappable text-to-speech module is used to speak each response back to the caller.

• Calls took too long to resolve. Checking an order or finding the correct product involved a slow back-and-forth, and each minute spent on the line meant that a customer was waiting rather than making a purchase.
• Concurrency broke the phone system. Traditional call handling was unable to handle many callers at the same time without having to greatly increase the number of staff and costs — the peaks in demand were simply too much for it to cope with.
• Scripted IVR menus frustrate callers. Rigid press-one decision trees are unable to carry on a natural conversation or give an answer to an open-ended product question as a person can.
• Answers had to reflect reality. Any suggestion regarding a product or update to an order had to be based on the current catalog and actual order data and not invented by the model.

The agent operates as a collection of concurrent and separate workers — the data flows from the incoming phone call, first through the telephony component and then via a real-time WebRTC worker, into grounded LLM reasoning and function calling, and then back to the caller by voice.

• Spec before code. In accordance with the SoftmaxAI Method, we prepared and approved a specification that outlined all the conversation flows, the order-lookup integrations, and the way in which the recommendations had to be based on the actual catalog before writing a single line of production code.
• Worker-per-caller isolation. When an inbound call comes in, it is assigned a specific agent worker within a separate LiveKit room, which ensures that the concurrent conversations remain independent, reliable, and easy to reason about when under load.
• Grounded reasoning, live actions. By referencing the product catalog, retrieval-augmented generation ensures that answers are based on factual information, while real-time function calling retrieves the order status when required, thus keeping the responses factual rather than invented.
• Human-owned quality. The senior engineers were in charge of the design and quality decisions, and they assessed the RAG grounding and function calling against realistic call scenarios until the behavior had become reliable before the system went live.

• Natural conversation over a normal phone line. When customers dial a regular number, they talk normally; the agent then listens to them, understands what they say, and responds to them by speaking rather than by sending them through a menu tree.
• Self-service order status. The agent checks the order-status API in real time as the conversation is taking place and responds to the question 'where is my order' without involving a human.
• Catalog-grounded product recommendations. Retrieval-augmented grounding compares the caller's stated need with the current product catalog so that the recommendations correspond to the products that are actually available.
• Reliable concurrency by design. Because each caller is assigned their own isolated worker, a large number of customers can be served at the same time without the conversations interfering with one another or deteriorating when there is a high volume of callers.
• A swappable, future-proof voice stack. Since speech-to-text and text-to-speech are modular in design, the voice or supplier can be changed without the need to re-engineer the conversation logic.