Listening through speaking in a single model
Voice AI company ElevenLabs on Oct. 20 released "Eleven Realtime v1," a new model for AI agents that converse by voice. @aitrackerbot, an X account that tracks the appearance of models, reported that it appeared in official documentation the same day (same day Japan time).
Until now, voice agents were often built by developers stringing together a component that turns speech into text, a language model that works out the answer and a component that turns text into speech. Eleven Realtime tunes those three, along with deciding when to take turns speaking, together as a single model. Speech goes in and speech comes out, but the intermediate stages are kept as text, so what the agent heard and how it decided can be checked afterward. It supports more than 90 languages and over 17,000 voices.
Max for complex matters, Mini for everyday conversation
There are two versions. "Max" is aimed at matters requiring step-by-step reasoning, such as customer service and qualifying prospective customers. "Mini" is lower-priced and suited to AI assistants, study partners and conversational characters.
In an evaluation by third-party Artificial Analysis, accuracy on reasoning about questions posed by voice was 98.8% for Max and 75.9% for Mini. Max's completion rate on tasks using tools, at 87.1%, was also well above Mini's 60.3%. On the other hand, Mini scored 95.8% on the naturalness of conversation, including interruptions and verbal acknowledgments, slightly above Max's 94.1%. Time to first audio averaged 0.57 seconds for Max and 0.62 seconds for Mini. Artificial Analysis said no model on its leaderboard is both faster and more accurate than Max.
| Evaluation | Max | Mini |
|---|---|---|
| Voice-based reasoning (Big Bench Audio) | 98.8% | 75.9% |
| Completion rate on tasks using tools | 87.1% | 60.3% |
| Naturalness of conversation (Full-Duplex-Bench) | 94.1% | 95.8% |
| Human preference rating (points) | 998 | 988 |
| Time to first audio (seconds) | 0.57 | 0.62 |
Compatible with OpenAI's spec; priced by tokens and time
The API uses the same specification as OpenAI's voice conversation API, "Realtime API." Existing programs can migrate by changing only the endpoint and model name, and OpenAI's development software works as is. The only supported transport for now is WebSocket; WebRTC, used in web conferencing and elsewhere, is not supported. The model can also be selected in the company's own agent-building service "ElevenAgents" and deployed directly to phone, web and WhatsApp.
Pricing is $0.08 per minute of connected audio, plus the language model's token charges. Per 1 million tokens, Mini costs $0.2 for input and $6 for output, while Max costs $1.8 for input and $9 for output. Each plan includes an allowance for usage, but ElevenLabs described it as an estimate based on typical token usage, so heavy use of Max draws it down faster.
| Price (per 1 million tokens, dollars) | Max | Mini |
|---|---|---|
| Input | 1.8 | 0.2 |
| Output | 9 | 6 |
A run of speech model releases
Voice AI releases clustered in the same week. China's Ant Group released the weights of "Ming-3.0," a model for speech synthesis and other tasks, on Oct. 19, the day before, and France's Mistral AI announced its "Voxtral 3" speech models on Oct. 20.
