Benchmarks don't book sales meetings

People ask why we don't publish benchmark numbers for our duplex model. Short answer: we run them, and they don't tell us which model will do better on real calls.

what the benchmarks ask your model

full-duplex-benchsmall talk What's the best time to plant tomatoes? big bench audiopuzzles If you follow these instructions, do you return to the starting point? τ³-benchhelp desk Hi, I want to return all my gaming stuff.

Our model makes sales calls for businesses, including lead qualification, follow-ups and customer outreach. Each call has one task the business hired us to do: reach the right person, make the pitch, and book the next step. On the way, it has to say who it is and why it's calling, and read the person's state: free, busy, frustrated or in a rush. It has to handle objections, since “not interested” is often worth digging into, and ask the right follow-up questions. And it has to do this wherever the person picks up, a quiet office, a car, a noisy street.

The public duplex benchmarks test different things: small talk, reasoning over puzzles, and following policies and using tools in help-desk transactions. They're evaluated in controlled settings often with prerecorded speech or an LLM playing a customer who wants to be helped. They don't capture the unpredictable dynamics of real sales conversations. A model can top these benchmarks and still fumble a sales call.

Below are three popular benchmarks, what they test, and what they miss. Every sample is real. Press play.

Full-Duplex-Bench tests the mechanics of taking turns: pauses, backchannels (“mm-hmm”) and interruptions. The talk itself is general chat about tomatoes, puppies and black holes.

samples

1,226 across 9 subsets

turns

1-2 per sample

audio

16 kHz · TTS, plus recorded chats

scoring

Stop time, response type, reply relevance

interrupt · tomatoes → faucet interrupt · smart home → movies backchannel · global warming backchannel · black holes

sampleThe benchmark provides only the caller's audio. Your model fills the purple lane, and its responses and timing are evaluated. Examples from the v1.5 user-interruption and user-backchannel subsets.

What it misses

• Each sample is one or two caller turns, scored at a single moment. There is no goal to reach and nothing to finish. (v2 adds longer conversations led by an AI examiner.)
• It measures whether the model handles an interruption, not whether it keeps the conversation on track. Its interruptions jump to an unrelated topic, and answering the new topic counts as success. On a sales call, an interruption is often an objection. The agent has to address it, then steer back to the goal. The benchmark doesn't test that.

Big Bench Audio takes 1,000 reasoning puzzles from the text benchmark Big Bench Hard and reads them aloud using TTS voices. The model hears each question, gives one answer, and an LLM grades it.

questions

1,000 · 4 kinds × 250

turns

1 · question in, answer out

audio

TTS · 23 voices

scoring

LLM judge vs. the official answer

Example questions · press a card to hear the question

Try one · can you solve this spoken puzzle?

short · 3 moves long · 7 moves

your answer yes no

Answer first. Then watch the walk.

What it misses

• One turn, one answer. Real sales calls involve many turns, interruptions, overlapping speech and background noise.
• Nobody asks our agent to solve a maze. Solving a spoken puzzle doesn't tell us whether a model can qualify leads, handle objections or book meetings.

τ³-bench puts an agent on an airline, retail or telecom help desk. A simulated customer wants a refund or a rebooking. The agent has to follow policies and use the right tools to complete the task. Success is measured by the final state of the database.

voice tasks

278 · airline, retail, telecom

tools

13-16 per domain

caller

An LLM with a TTS voice · 8 kHz phone audio

scoring

Final database state, plus checks

graded actions · retail task 14

0 / 6called so far

Pass = final database state shows both returns refunded to the correct cards.

What it misses

• It's a transaction: the agent verifies information, looks up records and makes changes. Our calls are sales conversations. We have to reach the right person, read their state, handle objections, ask relevant follow-up questions and move towards a booking.
• The customer calls in. The caller is an LLM that wants to be helped. Our calls go to people who've expressed interest but weren't expecting a call at that moment, leads who've gone cold, or existing customers we haven't spoken to in a while. They may be apprehensive, and the agent has to know when to listen, give them space and make the conversation worth their time.
• It grades a database change, not a sales outcome. τ³-bench is the closest of the three, with longer conversations, interruptions and tool use. But completing a help-desk task doesn't tell us whether an agent can qualify a prospect or book a meeting.

τ-bench 2.0 (19 October 2026) adds outbound calls, called τ-Entity. The job is to capture one missing detail exactly from a simulated person who has it ready, such as a phone number or address. It is still a database field, not a sale.

Artificial Analysis combines four benchmarks into a single score for speech models. Each contributes 25%, with half the score coming from tool-calling tasks.

25%τ-VoiceSierra's help-desk tasks. An LLM plays the caller. Graded on the final database state.

25%Task successDid the model make the right final tool call on the Arena's live calls?

25%Arena preferenceHuman testers make live calls to two hidden models and choose which they prefer.

25%Big Bench AudioSingle-turn reasoning puzzles graded by an LLM.

The Arena is the closest to a real call: real people, live conversations and many turns. But the tester calls the agent with a task they want completed. Conversations where testers deviate from their assigned task are excluded from task success scoring. Instead, our calls reach people in different states and environments, and navigating that unpredictability is part of the job.

Even within the Arena, the model people prefer isn't necessarily the one that completes the most tasks successfully. On Artificial Analysis's leaderboard, Gemini 3.1 Flash Live (Minimal) ranks 1–3 on human preference but successfully completes 74.6% of tasks. Grok Voice 2.0 ranks 8–14 on preference but completes 94.6%.

Source: Artificial Analysis Speech Agent Arena, 28 October 2026. Rankings are estimated ranges reflecting statistical uncertainty.

1. Realistic test calls. Our team role-plays scenarios seen in production, from busy or frustrated prospects to wrong contacts and unexpected objections. Labellers identify turns where the model fails to handle the conversation correctly.
2. A per-turn judge. We replay flagged moments using the real conversation up to that point, then score the model's response against rubrics built from human evaluations. This helps us measure whether a new model fixes the failures we've identified.
3. Live customer calls. We validate performance on live calls by measuring business outcomes, such as booked meetings and disqualifications.

When a public benchmark reflects the realities of our calls, we'll publish our scores.

We're also exploring releasing a benchmark for the community.

For now, we're focused on delivering business outcomes for our customers and pilot partners, like our lead qualification results with Polyborg, where Familiar matched the performance of a human SDR.