NativeCall quality report · Updated October 3, 2026 UTC
Translation quality, playback timing and speech recognition
We check whether translations retain names, dates and numbers, and measure when playback starts. Each study below includes its date, sample and method.
3 October 2026 UTC · Controlled text checks
Checking names, dates and numbers
117 / 120
Retained all specified critical facts
6
Directions: English, Mandarin and Cantonese ↔ Japanese
We checked 120 fixed synthetic telephone phrases, 20 per direction, for names, dates, times, amounts, quantities and negation. This is a diagnostic text-translation study.
Complete results and scoring
All 120 requests returned output. Agent review found 112 without a meaning issue, five with a minor wording issue and three with a critical fact or meaning error. The 117 critical-fact passes include the five minor wording cases; these counts overlap.
The three critical cases concern a duplicate-charge refund request, a surname and the JPY currency. All results are retained. The protected diagnostic route uses a controlled fact-preservation context. Review was by a Codex agent.
Recent real calls · Server timing
Time until translation starts playing
Toward the phone recipient · median
2.87 s
p95 8.87 s · 36 turns
Toward the app caller · median
3.38 s
p95 12.02 s · 32 turns
Measured from the system’s speech-end event to server playback start. Later network, device and acoustic arrival time are excluded.
Dates, sample and missing measurements
Collected 3 October 2026 UTC from the preceding 168 hours: 31 real translated calls with 420 stored timing turns. This measure was available for 68 turns; 352 lacked valid timing. Calls were found through Worker telemetry; calls without timing data are excluded.
Japanese speech-recognition experiment · August 23, 2026
Historical control C
15.29% CER
Character error rate; lower is better
Candidate D
2.61% CER
Character error rate; lower is better
Measurement scope: August 23, 2026; 5 unique Japanese clips, each repeated 3 times, with an 8 kHz telephone-codec simulation.
Read all four languages’ results and methodology ↓Historical call playback logs · August 29–September 1, 2026
Time until translation playback starts
Toward the app caller · median
1.36 seconds
Toward the app caller · p95
4.276 seconds
These 33 turns have both timestamps, measured from the recorded end of speech to the server’s translation playback-start event. The measurement covers the translation and playback path after recognition, making it closer to the call flow than recognition timing alone.
Measurement boundary and sample details
The previously selected sample contains 10 calls and 115 retained timing turns: 34 have this measurement and 81 lack one or both timestamps. Thirty-three measured turns are toward the app caller. There is one recipient-direction turn, at 1.315 seconds, so no percentile is calculated for that direction. The event ends at server playback start; it excludes later device, network and acoustic-arrival time.
Test configurations use anonymous group labels. Provider and model names are not published; these labels distinguish historical experiment configurations.
Historical speech-recognition experiment: all tested results
Each language has only 5 unique public FLEURS clips, each repeated 3 times: 15 trials per model and language.
The candidate did not improve every language: it had lower error rates on these Japanese and English clips, but higher error rates on Mandarin and Cantonese. All eight result rows are included below; the best numbers from different models are not combined into a single service score.
| Language / test group | Error rate | Completed trials | ASR p50 | ASR p95 |
|---|---|---|---|---|
| English · Historical control A | 7.50% WER | 15/15 | 279 ms | 411 ms |
| English · Candidate D | 6.67% WER | 15/15 | 484 ms | 664 ms |
| Mandarin · Historical control B | 5.52% CER | 15/15 | 241 ms | 330 ms |
| Mandarin · Candidate D | 6.21% CER | 15/15 | 277 ms | 625 ms |
| Cantonese · Historical control B | 2.34% CER | 15/15 | 225 ms | 323 ms |
| Cantonese · Candidate D | 3.13% CER | 15/15 | 193 ms | 489 ms |
| Japanese · Historical control C | 15.29% CER | 15/15 | Not comparable | Not comparable |
| Japanese · Candidate D | 2.61% CER | 15/15 | 208 ms | 433 ms |
Method and measurement scope
WER is word error rate; CER is character error rate. Both divide the total insertions, deletions and substitutions by the number of reference words or characters; lower is better. English uses words; Japanese, Mandarin and Cantonese use characters.
Scoring normalizes Unicode, punctuation, Traditional/Simplified Chinese and spoken/Arabic numeral formatting. Some writing differences therefore do not count as errors.
Audio underwent an 8 kHz mono G.711 mu-law encode/decode roundtrip and was streamed in real time in 100 ms frames. No task scripts, corpus prompts or hotwords were supplied. p50 is the median; p95 is the 95th percentile. Timing starts when audio submission ends and stops when the recognition provider reports completion. Translation, speech synthesis and playback are excluded.
The control C harness adds a fixed waiting period and has no comparable completion event, so its completion delay is withheld. “Control” refers to the historical experiment configuration.
Sources and related pages
Google FLEURS · Download the sanitized benchmark JSON
The aggregate records the date, anonymous test groups, sample sizes, scoring method and SHA-256 digest of the source score file.
Watch app translation demos and read the excerpts →
See task reporting and our completion standard →