Hosting = where inference runs; company = where the provider or lab is registered.
EU hosting means the route's inference runs inside the EU: an EU region, AWS Bedrock's EU cross-region (geo) profiles, Azure's Europe Data Zone, or a provider whose entire public fleet is documented as EU-hosted, each checked per model against the provider's documentation. Global deployments do not count, and neither does an EU billing region, an EU company or an EU control plane on its own. One disclosed company-policy exception stays in, marked “EU equivalent”.
Evidence requirements (benchmark evidence, priced provider, measured task tokens) sit above the table they apply to.
Applies to price views & model offers; benchmark evidence stays unfiltered.
Audio benchmark · v0.1 · 29 Sep 2026
Which audio models can make the decisions a voice agent needs — straight from the caller's audio, with honest probabilities, fast and cheaply?
17 systems · 988 scored items (112 public + 876 sealed) · intent, escalation, command safety, sentiment, urgency, speaker verification and sound events · only system-level aggregates are published; sealed items stay private.
Text decisions: JevBench · Image decisions: Image JevBench · How it is measured
Capability is the mean of Intelligence and Calibration. Qwen3-Omni-30B-A3B-Instruct leads the Jev-class audio systems with 83.2.
Jev-class: median latency ≤ 1.0 s and ≤ $0.25 per 1,000 decisions. 7 of 9 full-coverage systems meet the ranking gate. 1 additional provisional or partial row meets the speed/cost limits but is not ranked.
Ranked · full coverage (sealed + public items)
Not ranked · provisional and partial rows
Hosted APIs are scored on the 112 public items only, because sealed items never go to third-party APIs; the whisker is the 95% interval of their Capability. Sound classifiers answer only the sound-event family. Neither is comparable with the ranking above. Shown regardless of the Jev-class filter.
# numbers only Jev-class systems with full coverage. Right column: USD per 1,000 decisions (* = estimated from a catalogue price; see method) and adjusted median latency.
The two charts are linked: pointing at a system highlights it in both. The five most capable ranked Jev-class systems are labelled. Jev-class filter on; untick it above to add every system.
Median latency (adjusted p50, the Jev-class measure), log scale, faster to the right. The dashed line is the Jev-class limit (1.0 s). Upper right is better.
USD per 1,000 decisions, log scale, cheaper to the right. The dashed line is the Jev-class limit ($0.25). Upper right is better.
The official AudioJevBench score (option B): a weighted harmonic mean of Intelligence 40, Calibration 20, Speed 20 and Cost 20, with the JevBench v1.5 low-axis gates. Every full-coverage system is ranked here, including those outside Jev-class; a weak axis pulls the score down hard, and the row says which one.
Not ranked · provisional and partial rows
Same formula on what these systems could be measured on; not comparable with the ranking above.
Pick two systems and compare every published aggregate side by side. The better value in each row is bold.
Qwen3-Omni-30B-A3B-Instruct
self-hosted GPU · full coverage · composite #1
Qwen2.5-Omni-7B
self-hosted GPU · full coverage · composite #2
Up to four 0–100 axes; farther from the centre is better. Public-only rows use public Intelligence, and unmeasured axes are omitted.
All 17 measured systems in three labelled groups. Only the full-coverage group is ranked. Public-only rows show their public Intelligence with a stratified 95% bootstrap interval. Scroll sideways on small screens.
| # | System | Composite (B) | Capability | Intelligence | Public I (95% CI) | Sealed I | Gap · penalty | Calibration | Speed | Cost | p50 / p95 adj. | $/1k | Items | Jev-class |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full coverage (sealed + public) · 9 · ranked by composite | ||||||||||||||
| 1 | Qwen3-Omni-30B-A3B-Instructself-hosted GPU | 70.4 | 83.2 | 81.0 | 86.9 | 79.1 | 7.7 · ×1.00 | 85.4 | 82.4 | 49.2 | 660 ms / 880 ms | $0.049* | 988 | yes |
| 2 | Qwen2.5-Omni-7Bself-hosted GPU | 52.1 | 68.1 | 57.8 | 58.2 | 57.6 | 0.6 · ×1.00 | 78.5 | 80.7 | 46.1 | 676 ms / 1.26 s | $0.063* | 988 | yes |
| 3 | Qwen2.5-Omni-3Bself-hosted GPU | 48.2 | 53.8 | 50.1 | 50.8 | 49.9 | 0.9 · ×1.00 | 57.4 | 83.8 | 46.7 | 522 ms / 794 ms | $0.060* | 988 | yes |
| 4 | Voxtral Mini 3B (2507)self-hosted GPU | 36.6 | 47.8 | 47.2 | 41.7 | 49.0 | -7.3 · ×1.00 | 48.4 | 85.4 | 44.7 | 383 ms / 753 ms | $0.070* | 988 | yes |
| 5 | Whisper large-v3-turbo → Qwen3-4B-Instruct-2507cascade: transcript, then text model · self-hosted GPU | 19.3 | 57.6 | 48.2 | 44.6 | 49.4 | -4.7 · ×1.00 | 67.1 | 61.4 | 32.8 | 8.04 s / 8.97 s | $0.17* | 988 | no |
| 6 | MOSS-Audio-4B-Instructself-hosted GPU | 12.7 | 27.9 | 31.8 | 29.8 | 32.4 | -2.7 · ×1.00 | 23.9 | 85.4 | 46.5 | 396 ms / 731 ms | $0.061* | 988 | yes |
| 7 | AudioJev (mocomoco edge ONNX)local CPU | 0.0 | 0.0 | 0.0 | -55.2 | -55.5 | 0.3 · ×1.00 | 0.0 | 73.8 | 77.1 | 1.65 s / 2.52 s | $0.0058* | 988 | no |
| 8 | Qwen2-Audio-7B-Instructself-hosted GPU | 0.0 | 12.8 | 0.0 | -42.4 | -49.5 | 7.1 · ×1.00 | 25.6 | 81.8 | 38.6 | 653 ms / 1.00 s | $0.11* | 988 | yes |
| 9 | Ultravox v0.5 (Llama 3.2 1B)self-hosted GPU | 0.0 | 31.4 | 0.0 | -44.0 | -29.3 | -14.7 · ×1.00 | 62.9 | 91.1 | 61.4 | 224 ms / 347 ms | $0.019* | 988 | yes |
| Public-only, provisional · 6 · not ranked | ||||||||||||||
| – | Gemini 3.8 FlashAPIGemini API · public items only | 0.1 | 94.4 | — | 97.0 [91.9, 99.9] | — | — | 91.9 | 74.1 | 4.5 | 1.65 s / 2.37 s | $1.53 | 112 | no |
| – | Gemini 3.5 FlashAPIGemini API · public items only | 1.1 | 83.1 | — | 87.2 [78.9, 94.5] | — | — | 79.0 | 76.7 | 9.1 | 1.27 s / 1.69 s | $1.07 | 112 | no |
| – | Gemini 3.1 Flash-LiteAPIGemini API · public items only | 19.1 | 72.1 | — | 80.1 [70.9, 88.2] | — | — | 64.0 | 77.2 | 29.0 | 1.27 s / 1.51 s | $0.23 | 112 | no |
| – | Gemini 3.5 Flash-LiteAPIGemini API · public items only | 19.4 | 65.6 | — | 73.5 [62.6, 83.9] | — | — | 57.8 | 76.5 | 29.7 | 1.32 s / 1.69 s | $0.22 | 112 | no |
| – | GPT Audio MiniAPIOpenAI API · public items only | 0.5 | 66.3 | — | 68.6 [57.9, 79.6] | — | — | 64.0 | 79.8 | 7.2 | 844 ms / 1.24 s | $1.24 | 112 | no |
| – | GPT AudioAPIOpenAI API · public items only | 0.0 | 69.3 | — | 67.3 [56.1, 77.9] | — | — | 71.3 | 78.3 | 0.0 | 947 ms / 1.55 s | $4.94 | 112 | no |
| Partial coverage · 2 · not ranked | ||||||||||||||
| – | CLAP (laion/clap-htsat-fused)local CPU · sound events onlycovers: sound event | 0.0 | 47.5 | 95.1 | 100.0 | 93.4 | 6.6 · ×1.00 | 0.0 | 81.7 | 96.3 | 613 ms / 1.11 s | $0.0013* | 95 | yes |
| – | AST (MIT AudioSet fine-tune)local CPU · sound events onlycovers: sound event | 0.0 | 29.9 | 59.8 | 37.5 | 67.2 | -29.7 · ×1.00 | 0.0 | 68.1 | 98.5 | 3.94 s / 3.96 s | $0.0011* | 95 | no |
Accuracy per condition slice (Score items count 1 − normalised error). Diagnostics only — not part of any axis. Small n means wide uncertainty; public-only rows cover public items only.
| System | Clean audio | Noisy (≤ 5 dB SNR) | 15 dB | 10 dB | 5 dB | 0 dB | English | Non-English | Background talker | Overlapping talkers | Speaker verification | Sound events | Noise drop (clean − ≤5 dB) | Language gap (EN − non-EN) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-Omni-30B-A3B-Instructself-hosted GPU | 87% n=691 | 89% n=125 | 92% n=82 | 93% n=89 | 90% n=89 | 83% n=23 | 89% n=633 | 88% n=355 | 84% n=112 | 74% n=58 | 64% n=116 | 90% n=179 | -1 pts | +1 pts |
| Qwen2.5-Omni-7Bself-hosted GPU | 77% n=691 | 76% n=125 | 83% n=82 | 85% n=89 | 74% n=89 | 79% n=23 | 80% n=633 | 75% n=355 | 74% n=112 | 76% n=58 | 51% n=116 | 90% n=179 | +1 pts | +6 pts |
| Qwen2.5-Omni-3Bself-hosted GPU | 74% n=691 | 78% n=125 | 71% n=82 | 74% n=89 | 75% n=89 | 86% n=23 | 75% n=633 | 73% n=355 | 71% n=112 | 66% n=58 | 49% n=116 | 82% n=179 | -3 pts | +2 pts |
| Voxtral Mini 3B (2507)self-hosted GPU | 70% n=691 | 72% n=125 | 84% n=82 | 77% n=89 | 73% n=89 | 82% n=23 | 69% n=633 | 77% n=355 | 77% n=112 | 69% n=58 | 49% n=116 | 50% n=179 | -2 pts | -8 pts |
| Whisper large-v3-turbo → Qwen3-4B-Instruct-2507cascade: transcript, then text model · self-hosted GPU | 65% n=691 | 75% n=125 | 82% n=82 | 82% n=89 | 78% n=89 | 82% n=23 | 65% n=633 | 76% n=355 | 78% n=112 | 74% n=58 | 22% n=116 | 37% n=179 | -10 pts | -11 pts |
| MOSS-Audio-4B-Instructself-hosted GPU | 63% n=691 | 59% n=125 | 66% n=82 | 61% n=89 | 54% n=89 | 75% n=23 | 65% n=633 | 58% n=355 | 58% n=112 | 59% n=58 | 50% n=116 | 67% n=179 | +5 pts | +7 pts |
| AudioJev (mocomoco edge ONNX)local CPU | 9% n=691 | 5% n=125 | 8% n=82 | 12% n=89 | 4% n=89 | 5% n=23 | 9% n=633 | 8% n=355 | 5% n=112 | 5% n=58 | 0% n=116 | 13% n=179 | +4 pts | +1 pts |
| Qwen2-Audio-7B-Instructself-hosted GPU | 10% n=691 | 6% n=125 | 13% n=82 | 14% n=89 | 5% n=89 | 5% n=23 | 13% n=633 | 5% n=355 | 7% n=112 | 9% n=58 | 0% n=116 | 10% n=179 | +4 pts | +7 pts |
| Ultravox v0.5 (Llama 3.2 1B)self-hosted GPU | 16% n=691 | 15% n=125 | 18% n=82 | 28% n=89 | 16% n=89 | 12% n=23 | 17% n=633 | 18% n=355 | 10% n=112 | 7% n=58 | 2% n=116 | 19% n=179 | +1 pts | -1 pts |
| Gemini 3.8 FlashAPIGemini API · public items only | 97% n=80 | 100% n=10 | 100% n=10 | 100% n=12 | 100% n=6 | 100% n=2 | 99% n=72 | 97% n=40 | 100% n=14 | 100% n=12 | 95% n=20 | 96% n=24 | -3 pts | +1 pts |
| Gemini 3.5 FlashAPIGemini API · public items only | 89% n=80 | 100% n=10 | 90% n=10 | 100% n=12 | 100% n=6 | 100% n=2 | 92% n=72 | 90% n=40 | 93% n=14 | 92% n=12 | 65% n=20 | 96% n=24 | -11 pts | +2 pts |
| Gemini 3.1 Flash-LiteAPIGemini API · public items only | 82% n=80 | 100% n=10 | 100% n=10 | 100% n=12 | 100% n=6 | 100% n=2 | 90% n=72 | 82% n=40 | 71% n=14 | 67% n=12 | 60% n=20 | 95% n=24 | -18 pts | +8 pts |
| Gemini 3.5 Flash-LiteAPIGemini API · public items only | 79% n=80 | 100% n=10 | 87% n=10 | 92% n=12 | 100% n=6 | 100% n=2 | 89% n=72 | 72% n=40 | 79% n=14 | 75% n=12 | 55% n=20 | 88% n=24 | -21 pts | +17 pts |
| GPT Audio MiniAPIOpenAI API · public items only | 77% n=80 | 99% n=10 | 88% n=10 | 80% n=12 | 99% n=6 | 100% n=2 | 82% n=72 | 77% n=40 | 64% n=14 | 67% n=12 | 50% n=20 | 84% n=24 | -22 pts | +6 pts |
| GPT AudioAPIOpenAI API · public items only | 72% n=80 | 99% n=10 | 89% n=10 | 83% n=12 | 98% n=6 | 100% n=2 | 77% n=72 | 77% n=40 | 71% n=14 | 67% n=12 | 15% n=20 | 95% n=24 | -27 pts | -0 pts |
| CLAP (laion/clap-htsat-fused)local CPU · sound events only | 95% n=80 | 91% n=11 | — | 100% n=3 | 88% n=8 | — | 95% n=95 | — | — | — | — | 95% n=95 | +4 pts | — |
| AST (MIT AudioSet fine-tune)local CPU · sound events only | 85% n=80 | 82% n=11 | — | 100% n=3 | 88% n=8 | — | 85% n=95 | — | — | — | — | 85% n=95 | +3 pts | — |
Public items as the system receives them: an operator context, a typed question and the allowed labels, plus the audio clip (not embedded here). Gold is marked. Sealed items are never shown.
sound event · choice · 9.5 s clip
Context: You are the sound-event classifier inside the yard monitor of Redfern Smallholding, used at dawn. Label the main acoustic event in the clip.
Which event best describes the main sound in the clip?
car_alarm — a car alarm going offrooster — a rooster crowinggoldcar_horn — a car horn honkingsiren — a siren wailing (emergency vehicle or warning siren)intent routing · choice · 9.0 s clip
Context: You are the phone assistant of Pharmacie Exemple Demo. The caller is the first voice you hear; another person near the caller may talk at the same time. Route only the caller's request.
Which service does the caller need?
prescription_refill — Renew or prepare a repeat prescription.goldopening_hours — When the pharmacy is open.speak_to_pharmacist — Medical advice about a medicine or symptom.delivery_status — Where a home delivery is.urgency · 0–4 score · 12.5 s clip
Context: You are the dispatcher assistant of Soccorso Stradale Demo, a roadside assistance club. Assign the dispatch priority defined in the question from the member's description.
Which dispatch priority applies?
0 — 0 - Information only (membership, coverage questions); no dispatch.1 — 1 - Book a workshop appointment; the car is still drivable.2 — 2 - Dispatch within 4 hours: the car will not drive but is at home or in a car park or other safe place.gold3 — 3 - Dispatch within 1 hour: stranded at the side of an ordinary road, people are safe.4 — 4 - Immediate priority: stopped on a motorway or in a traffic lane, smoke or fire, or vulnerable people (children, elderly, ill) exposed to danger.caller escalation · yes/no · 11.2 s clip
Context: You are the repairs line assistant of Willow Court Housing Demo. Policy: if there are signs of immediate danger in the caller's home (fire or smoke, a smoke alarm going off because of a real hazard, gas, sparking electrics, water reaching electrics), transfer to the emergency repairs line. Ordinary repairs are booked by you. Sounds from the caller's surroundings may be audible.
Should this call be transferred to the emergency repairs line?
yes — There are signs of immediate danger in the home.no — It is an ordinary repair with no sign of immediate danger.goldspeaker verification · yes/no · 19.4 s clip
Context: Voice ID check for Hilltop Federal Savings. The first part of the clip, before the beep, is the enrolment recording of the account holder made at sign-up. After the beep is the current caller.
Is the current caller the same person as the enrolled account holder?
yes — The probe voice (after the beep) is the same speaker as the enrolment voice (before the beep).goldno — The probe voice (after the beep) is a different speaker from the enrolment voice (before the beep).sentiment · choice · 7.8 s clip
Context: You are the post-delivery survey assistant of Tiendita Sample, an online shop. Tag the customer's spoken answer with one label.
Which label best describes the customer's answer?
satisfied — Positive about the order or delivery.neutral — Neither positive nor negative; the delivery was simply as expected.goldfrustrated — Negative about the order or delivery, but no statement about leaving.churn_risk — Says they will stop buying here, cancel, or switch to another shop.command safety · yes/no · 6.0 s clip
Context: You are the in-car assistant of Northwind Motors Example Co. The car is currently moving at 80 km/h. While the car is moving, you may change climate, navigation, music and phone calls, but you must not play video or open the web browser on the centre screen.
Should the assistant execute the driver's request now?
yes — The request is allowed while the car is moving.no — The request is blocked while the car is moving.goldWhat it measures. Each item is a 16 kHz mono clip plus a short operator context and a typed question. The decisive evidence is only in the audio. The system answers with a probability for every allowed label: a choice among named options, a yes/no, or a 0–4 score — the same contract as JevBench. Decision families: speech intent routing, escalation to a human, command safety, caller sentiment and urgency, overlapping and background talkers, speaker verification (enrolment sample vs caller) and sound events (alarms, horns, animals, household sounds), under clean and noisy conditions (15 to 0 dB SNR) and in 13 languages and 12 English accents.
Public and sealed. 112 public items (authored separately, commercially usable voices) and 876 sealed items are scored; a pre-registered audio-quality gate removed a further 6 public and 18 sealed items for every system alike. Sealed items are never published and go only to systems we run ourselves, on our own rented GPUs (offline container, no gold, pod removed afterwards) or on this server. Intelligence blends 25% public / 75% sealed (the public set is about an eighth of the items); a public-minus-sealed gap beyond 15 points above the field median costs Intelligence.
Hosted APIs are public-only. Sealed items are never sent to third-party APIs: no enforced zero-retention route was available for these providers. Gemini and OpenAI models are therefore scored on the public items only, shown as provisional with a 95% bootstrap interval, and never ranked with the full-coverage systems. Their numbers can move once a sealed-safe route exists.
Partial coverage. CLAP and AST are sound classifiers: they answer only the sound-event family (choice), so their Intelligence covers that family alone. Their label softmax is not treated as calibrated, so Calibration is 0.
Axes and scores. Four axes, 0–100: Intelligence (tier- and type-weighted accuracy, chance-corrected; missing or invalid answers count wrong), Calibration (are the probabilities honest), Speed (100 − 20·log10(latency / 0.1 s), mean of p50 and p95; self-hosted latency gets the JevBench ×2 + 0.15 s adjustment) and Cost (100 − 30·log10(USD per 1,000 / 0.001)). Capability = mean(Intelligence, Calibration). The composite (option B) is a weighted harmonic mean 40:20:20:20 with the JevBench v1.5 gates.
Jev-class gate. Adjusted median latency ≤ 1.0 s and ≤ $0.25 per 1,000 decisions, fixed before any result because there is no audio Jev reference model. It decides who gets a Capability rank number; the composite ranks every full-coverage system.
Pricing basis. Hosted rows use the provider's standard list price in force for at least 30 days, applied to the measured audio and text tokens; promotional prices with an end date do not count, so Gemini 3.8 Flash is priced at its announced regular rate. Self-hosted rows (marked *) are estimates: the exact model's public catalogue price if one exists, otherwise its base model's or the nearest same-family size (erring high), applied to the tokens counted by the model's own processor. Classifiers use the JevBench size-class rule for small encoders.
Limits. Speech clips are synthetic (text-to-speech voices with added real and synthetic noise); sound events are real recordings. One run per system at temperature 0. A system at Intelligence 0 answered at or below chance after correction, often by refusing or returning invalid output. Results describe these exact configurations.
7f78d3ee…d50b), inheriting the JevBench v1.5 method and pricing rules; two scorer implementation errata logged before scoring (gate exclusions, public-only scoring), no formula change. 17 systems: 9 full coverage, 6 public-only, 2 partial.AudioJevBench is a separate benchmark from the text-only JevBench and from Image JevBench. Sealed item content stays private; this page shows system-level aggregates only.