Updated October 30, 2026
AI voice agents now answer real phone calls. They book appointments, take orders, and route callers to the right team. Building one has become easy: a few hours with a voice agent platform can produce a working demo. However, knowing whether it is good enough for real callers is much harder. A successful demo call reveals very little about how the agent will handle the next thousand calls. AI voice agent evaluation metrics help teams assess performance, spot issues, and decide whether an agent is ready for real-world use.
This guide explains eight essential metrics, what each one measures, and how to evaluate performance using real numbers from open benchmarks to understand what good results look like.
Understanding AI Voice Agent
An AI voice agent is software that holds a spoken conversation and acts on it.
Most voice agents follow a three-step loop:
- Speech-to-text (STT) turns the caller’s audio into text.
- A large language model (LLM) decides what to say and which tools to call, such as a booking system or a CRM.
- Text-to-speech (TTS) turns the reply back into audio.
Newer speech-to-speech models handle audio in and audio out in a single model. Either way, the agent must also manage the conversation: notice when the caller has finished speaking, stop when it is interrupted, and pass the right details to other systems. A voice agent can fail at any of these steps, which is why one score is never enough. The eight metrics below cover the full call.
8 Essential AI Voice Agent Evaluation Metrics
The following eight metrics help teams understand how effectively an AI voice agent handles conversations and completes tasks.
1. Task Completion Rate
Task completion rate is the share of calls in which the agent achieves what the caller needed: the appointment is booked, the order is placed, or the caller reaches the right department. It is the closest thing to a single “did it work?” number.
- How to measure: write down the expected outcome for each test scenario, run the call, and grade whether the outcome happened. Teams usually grade with an LLM judge, checked against a sample of human reviews.
- What good looks like: when the same agent ran on 9 voice agent platforms in Cekura’s open voice AI benchmarks, task completion ranged from 87.8% to 98.8%.
2. Consistency Across Repeated Runs
LLMs are not deterministic. The same agent can pass a scenario on one run and fail it on the next, so a single test run flatters it. Measure consistency by running each scenario several times and counting only the scenarios that pass every time. With three runs, this is called pass³ (“pass to the power of three”).
- How to measure: run each scenario at least three times and report the share that passed all three.
- Why it matters: in the benchmark above, the top platform completed 98.8% of calls, but passed 87.8% of scenarios on all three runs. Another configuration completed 87.8% of calls but passed only 30.5% of scenarios every time. Consistency is often where agents that looked ready fall apart.
3. Tool Call Accuracy
When a voice agent books an appointment, it sends data such as a name, date, and phone number to another system through a tool call. Tool call accuracy checks that the data sent matches what the caller said and that the right tool was called at the right moment.
- How to measure: compare each tool call with the expected values for the scenario. This can be a simple, deterministic check that needs no AI judge.
- Why it matters: in one benchmark call, the transcript showed the caller’s phone number correctly, but the agent sent a different number to the booking tool. The conversation sounded perfect, and the booking was still wrong. Reading transcripts alone would never catch this.
4. Response Latency
Latency is the gap between the caller finishing a sentence and the agent starting to reply. In conversation, people notice pauses quickly, and long gaps make callers repeat themselves or hang up.
- How to measure: Time each turn from the end of the caller’s speech to the start of the agent’s audio. Report the median and the slow tail (the 90th or 95th percentile), not just the average, because a few very slow replies do the most damage.
- What good looks like: Across the 9 platforms, mean reply time ranged from 1.27 to 3.08 seconds. The fastest speech-to-speech model in a separate test replied in a median of 1.34 seconds.
5. Turn-Taking (End-of-Turn Detection)
Turn-taking is the agent’s ability to tell a pause from the end of a sentence. Reply too early, and the agent talks over a caller who was still thinking. Reply too late, and the call feels slow.
- How to measure: Play recordings that include mid-sentence pauses, filler words (“uh…”), and finished turns, then score how often the agent waits during pauses and replies after finished turns. Test on phone-quality audio and with background noise, not only clean recordings.
- What good looks like: In a turn-taking benchmark of 13 real-time models, the typical model replied to 99% of finished turns but waited through only 73% of pauses. Noise made things much worse: the best model in a quiet room answered 97% of finished turns, but only 55% in heavy noise.
6. Interruption Handling
Callers interrupt all the time to correct a detail or skip ahead. A good agent stops talking when it is interrupted but keeps going when the caller only says “mm-hmm” to show they are listening.
- How to measure: Inject interruptions and short acknowledgments at different points in the agent’s reply, then check whether it stopped or continued correctly and whether it recovered smoothly.
- What good looks like: In the same turn-taking benchmark, the typical model stopped for 100% of real interruptions but kept talking through only 6% of quick “mm-hmm”s. In other words, most models stop for any sound, which makes conversations choppy.
7. Speech Accuracy (Hearing and Saying it Right)
Speech accuracy covers both ends of the call: how well the agent hears the caller, and how well it speaks numbers, codes, and names.
- How to measure what it hears: Use word error rate (WER), the number of substituted, deleted, and inserted words divided by the number of words actually spoken. Lower is better. Pay special attention to names, dates, and numbers, because one wrong digit can break a booking.
- How to measure what it says: Have the agent read hard sentences (prices, tracking codes, medical terms), then transcribe the audio and check every item was read correctly.
- What good looks like: Among 19 streaming speech-to-text models, the lowest overall WER was 2.38%. Real phone calls are harder than clean test clips. One model’s WER rose from 3.9% on public clips to 12.5% on a small set of 8 recorded real calls. For text-to-speech, the most accurate voice read 93.4% of the hard items correctly.
8. Call Reliability
Call reliability measures whether the call works: the line connects, providers respond, and the agent never leaves the caller in silence.
- How to measure: Track the share of calls with no connection or provider errors, and the share with long silences (for example, 10 seconds or more with no reply). Keep failed calls in your results rather than discarding them.
- What good looks like: Across the 9 platforms, the share of calls with no infrastructure problem ranged from 72.4% to 100%. Among 12 speech-to-speech models, calls where the agent went quiet for 10 seconds or more ranged from 0% to 12.2%.
Side-by-Side Summary
| Metric | What it Tells You | How to Measure | Watch Out For |
| Task Completion Rate | Did the caller get what they needed? | Expected outcome per scenario, graded by an LLM judge | Judges must be checked against human reviews |
| Consistency (Pass³) | Does it work every time? | Run each scenario three times | A single run hides flaky behavior |
| Tool Call Accuracy | Did the right data reach your systems? | Compare tool inputs with expected values | Perfect transcripts can hide wrong data |
| Response Latency | Does the agent reply quickly? | Time per turn; median and 95th percentile | Averages hide slow outliers |
| Turn-Taking | Does it wait for the caller to finish? | Pauses and fillers on phone-quality audio | Noise sharply reduces accuracy |
| Interruption Handling | Does it stop when cut off? | Inject interruptions and “mm-hmm”s | Stopping for every sound |
| Speech Accuracy | Does it hear and say details right? | WER, plus reading tests for numbers and codes | Clean clips overstate real-call accuracy |
| Call Reliability | Does the call work at all? | Connection errors and long silences | Discarding failed calls from results |
How to Measure AI Voice Agent Evaluation Metrics in Practice?
You do not need a large team to start. A practical first evaluation looks like this:
- Create 20 to 50 test scenarios: Use real or expected customer interactions, including challenging situations such as noisy phone lines, interruptions, and callers changing their requests.
- Simulate different callers: Use a second AI agent to represent callers with different accents, speaking speeds, and background noise conditions.
- Run each scenario three times: Evaluate every call against all eight AI voice agent evaluation metrics to identify inconsistencies and recurring issues.
- Fix issues and repeat the tests: After changing a prompt, model, or configuration, run the entire test set again. This helps ensure that improvements in one area do not create problems elsewhere.
- Monitor performance after deployment: Continue tracking live calls because real customers may introduce situations that were not included in the original test scenarios.
Doing this by hand gets slow quickly. Testing platforms like Cekura automate this: they simulate thousands of callers against your agent, score each call on outcome, tool accuracy, latency, and interruptions, and monitor live calls once the agent is in production.
Final Thoughts
A voice agent that sounds great in a demo can still fail real callers in quiet, costly ways: a wrong digit in a booking, a long silence, or a reply that cuts the caller off. Measuring task completion alone will not show you these problems. Start with the eight AI voice agent evaluation metrics discussed in this guide, run every scenario more than once, and keep failed calls in your numbers. The agents that hold up in production are the ones that were tested like production.
Author Bio
Tarush Agarwal is co-founder and CEO of Cekura (YC F24), which tests and monitors voice and chat AI agents. Before Cekura, he built low-latency trading simulations in quantitative finance.
Frequently Asked Questions (FAQs)
Q1. What is the most important metric for an AI voice agent?
Answer: Task completion rate is the best single measure, but only when you also check consistency across repeated runs. An agent that completes a task once but fails it on the next run is not reliable.
Q2. What is a good response time for a voice agent?
Answer: Faster is better, but reliability matters too. In open benchmarks of 9 voice agent platforms, mean reply times ranged from 1.27 to 3.08 seconds. Track the slowest 5% of replies, not just the average.
Q3. What is word error rate (WER)?
Answer: Word error rate is the share of words a speech-to-text model gets wrong: substitutions plus deletions plus insertions, divided by the number of words spoken. A WER of 3% means about 3 errors per 100 words.
Q4. How many times should I run each test?
Answer: At least three. Because language models are not deterministic, the same scenario can pass and fail on different runs. Counting only the scenarios that pass every run (pass³) gives a truer picture.
Q5. Can I evaluate a voice agent using transcripts only?
Answer: No. Transcripts miss latency, interruptions, long silences, and the data sent to other systems. Measure the audio and the tool calls as well.
Recommended Articles
We hope this guide to AI voice agent evaluation metrics helps you test your agents with confidence. Explore these recommended articles for more insights:
