The Same AI Voice Ranks 3rd in English and 19th in Turkish

Summary

Most AI voice companies show you a demo page. Very few tell you how they tested, how confident they are, or how the voice holds up in a language other than English. We needed to pick a voice for a phone agent that talks to real patients, could not find that information anywhere, so we measured it ourselves.

Round 2026.2 covers 44 English voices and 28 Turkish voices from 16 companies. Two voices read the same sentence, a listener picks one, and nobody sees a brand name until voting closes. Every score ships with an error bar, and the full results file is a public download.

The format is not ours — blind pairwise voting has been the standard for years. Two things inside it are. Every vote came from a vetted panel of paid native speakers rather than from anyone who could reach the page, and we asked four separate questions rather than one. Both changed what came out, and the rest of this report is mostly the consequences.

What We Did Differently

Blind pairwise voting is a solved format and we did not reinvent it. Hugging Face's TTS Arena has run it for years: hide the brand, play two clips, count the wins, publish an Elo table. We kept all of that. We changed who votes and what they are asked.

Neither change is free. A closed paid panel fills slowly and stays small: 12,245 votes, not millions. Four questions split those votes four ways, so every error bar is wider than it would be if we pooled everything into one ranking. And a vetted panel is still not a representative sample of either language — people who work on AI training data listen more carefully than a patient on a phone call does.

We took both costs because the alternative is a number nobody can interpret: an unknown population answering an undefined question, updating live. We would rather publish a smaller frozen table and tell you exactly who produced it.

What Changed Since The Pilot

Our July pilot was honest about being too small: 2,675 votes across 40 English voices meant 21 of the 22 ranked entries overlapped the leader, and we said plainly that we could not rank English. Round two is 4.6 times bigger, and the picture is different in three specific ways.

First, English separates now — though not at the top. The error bars roughly halved, and 20 of the 44 voices are now measurably behind the leading pack instead of statistically tangled with it. The other 24 still overlap each other.

Second, the cross-language finding survived and got stronger. It was the one pilot result we said we believed, and more data has not softened it.

Third, and least comfortably, one pilot finding did not survive. We reported that the four listening questions measured genuinely different things, pointing at a near-zero correlation between “sounds human” and overall preference in Turkish. With four times the data that correlation is 0.89. It was noise. We were measuring our own sample size.

Splitting the questions is still the right call, and not only because it is how you find out they agree. Agreement in aggregate is not agreement per voice: the same English data has 33 of 44 voices moving ten or more places depending on the question, and the two questions that track each other least — “sounds human” and “clear and correct”, at 0.59 — are exactly the two a buyer has to trade off. Better At What goes through it. What we no longer claim is that the questions measure unrelated things, and anyone who quoted our pilot on that should stop.

How It Works

The point of the setup is to test voices, not brand loyalty or listener patience. Nothing about the method changed between rounds, which is what makes the two comparable.

Who does the listening

Listeners are paid contributors on joinvoicedata.com, and they only reach listening work after clearing the platform's onboarding — an approved voice recording, or an approved application. Most have done linguistics or AI training-data work before. Nobody judges a language they did not declare as native, and there is no open sign-up: the arena page has no vote button, so the panel is the panel. Round two was 224 people across 556 completed sittings, paid per comparison.

Both voices read the same sentence

Every comparison uses identical text, from a fixed bank of 24 sentences — 12 English, 12 Turkish — covering plain statements, questions, lists, names, numbers and dates, abbreviations, long winding sentences, and emotional lines.

Nobody sees the brand names

The listener's browser gets an audio file and its length, and nothing else. File names are random IDs, so no company or model name appears in the page, the network traffic, or anywhere a curious listener could dig. Buttons say “First voice” and “Second voice”. The map from clip to company lives in a separate file that is only opened after voting closes.

Four questions, asked one at a time

A listener gets 25 comparisons in a sitting, and all 25 ask the same one of four questions: which voice you prefer overall, which sounds more like a person, which is clearer and gets the words right, or which has the better rhythm and expression. They are never mixed inside a sitting, because an impression formed while judging rhythm should not decide the next answer about pronunciation. The server rejects any vote where the listener did not play at least 60% of both clips, so a modified browser cannot fake it. The two players cannot run at once. Replays are unlimited. There is no skip — you pick a voice, say they sound the same, or report an audio problem. Nobody sees the same pair twice, and nobody votes more than once on a given question in a given language.

Hidden checks

Three of the 25 comparisons are traps: the same clip played on both sides, at positions 5, 13 and 21. They look completely normal, both sides getting their own download link for the identical file. The honest answers are “about the same” or “audio problem”; picking a winner counts as a miss. Traps are paid like any other comparison, which is what keeps them hidden, and they never count toward anyone's score.

How the scores work

We use a standard head-to-head method called Bradley–Terry, which turns a pile of pairwise results into a strength number per voice. A tie counts as half a win. Scores are centred so the average voice in each language sits at 1000. Error bars come from resampling listeners rather than individual clicks, because one person voting 25 times is not 25 independent opinions. A voice needs 20 votes to earn a rank; in this round every voice cleared that easily.

What We Collected

Voting ran from 20 to 25 July 2026 and closed at 556 completed sittings.

English Results

This is what the pilot could not give us, with one caveat. The field is 44 voices, the bottom 20 are now measurably behind, and the top 24 are still a crowd.

Google Gemini's gemini-3.1-flash-tts-preview leads at 1138, but its range overlaps every voice down to rank 24. The honest reading is that the English field splits in two: a broad leading pack of 24 voices we cannot order internally, and 20 that are clearly behind it. That is a real gain over the pilot, where only one voice was separable from the leader — but it is separation of the bottom half, not a winner at the top.

Turkish Results

Turkish has a smaller field and a much clearer top. 19 of 27 challengers are clearly behind the leader.

The Language Gap

Sixteen of the voices were put in front of listeners in both languages — identical company, model and voice ID. That is the cleanest possible test of whether quality travels, because nothing changes except the language being read.

It does not travel.

One important exception, stated plainly: Google Gemini's gemini-3.1-flash-tts-preview Puck is ranked 1st in both languages. A voice can be good in two languages at once. The point is that most are not, and you cannot tell which from an English score.

Pooling all four questions per company gives each one enough comparisons for the cross-language picture to resolve sharply. This is the most useful thing in the report.

The two highlighted companies move in opposite directions, and neither move is small.

Cartesia repeats its pilot result almost exactly, which is the best evidence we have that the effect is real rather than an artefact of one round. Speechify is the new and starker case: first in English, eighth of ten in Turkish.

Six of the sixteen companies fielded nothing in Turkish at all — Smallest, Gradium, StepFun, Hume AI, Rime and Inworld. Turkish speakers choose from 28 voices against 44, and four of the top ten Turkish slots belong to Google.

Better At What?

This is the section that only exists because we asked four questions. Every number below comes from the same 12,245 votes; the only thing that changes is which question the listener was answering.

Start with the strongest case. In English, Fish Audio's s2-pro is 5th of 44 on sounding human and 40th of 44 on getting the words right. That is not a wobble inside the error bars: on “sounds human” its range overlaps the leader's, and on “clear and correct” its entire range (819–963) sits below the bottom of the leader's range at 1090. The same clips, the same listeners, the same week. It sounds like a person and it mispronounces things.

Across the whole English field, 33 of 44 voices move ten or more places between their best and worst question. The median voice moves 13 places; the widest moves 35. Ask one question and you get one of those columns, with no way to know which.

The two questions that disagree most are the two that matter most commercially. “Sounds human” and “clear and correct” track each other at 0.59 in English, against 0.81 for clarity versus rhythm. A demo reel is won on the first; a phone call that reads back an address is won on the second.

xAI owns one question, Google owns the others

The clearest company-level split in the data. xAI's altair is the outright top of the English “sounds human” board at 1155, with 26 of the 44 voices clearly behind it — and in Turkish, xAI's ara tops that same question at 1188. Whatever xAI is doing about sounding like a person, it travels across both languages we tested.

It does not win the rest. In Turkish, Google Gemini leads three of the four questions outright, and the single question it does not lead is the one xAI takes.

Being careful about that Turkish naturalness row: 68.0% and 64.5% have overlapping ranges, so read it as xAI edging ahead rather than beating Google. The voice-level result behind it is the harder evidence, and it points the same way in both languages.

The English winner makes the same point from the other side. Google Gemini's gemini-3.1-flash-tts-preview Puck is 1st overall and 2nd on rhythm, but only 11th of 44 on sounding human and 11th on clarity. It wins English on delivery, not on humanity. An arena that asked one question would have told you it is simply the best English voice, and that is not what the listeners said.

Two honest qualifications. First, in aggregate the four questions do broadly agree on the ordering — between 0.61 and 0.76 against overall preference in English — and we are not reviving the retired pilot claim that they measure unrelated things. The disagreement is per voice, which is unfortunately the level at which you have to choose one. Second, Turkish is much tighter than English: the median Turkish voice moves 6 places across the four questions and the widest moves 12, against 13 and 35 in English. With 28 voices and fewer votes, we would expect a smaller field to spread less; do not read the Turkish calm as a finding.

Voice Versus Model

People compare companies and models, but what you ship is one voice. Within a single model the spread between voices is still large enough to dominate the choice of model.

Fish Audio's English pairing is the extreme case: the same model, the same language, 333 points and ten ranks apart, with one of the two finishing last of all 44 voices. Test the voice you are going to ship, not the company that makes it.

Do Listeners Pay Attention

A listening test is only as good as the people doing the listening. Unlike the pilot, this round had the hidden traps running the whole time, so the answer is measured rather than assumed.

So roughly one time in ten, someone hears the exact same clip twice and still picks a favourite. That is the cost of asking humans to compare things, and it is why we put the traps in. Any listening test without checks like these is carrying about the same noise and does not know it.

Who We Excluded, And Why

Being transparent about this matters more than the result looking clean, so here is exactly what happened before we froze the numbers.

Two contributors tripped the automatic disqualification rule, which needs at least 6 controls seen, at least 3 misses, and a miss rate of 50% or higher. Their votes are out.

A further 51 sat on a softer hold, triggered by two misses. We looked at each one against the 9.3% population miss rate:

The 20 whose records are statistically consistent with honest listening were readmitted, returning 1,103 votes to the ranking. The 31 whose records are not — 23 of whom missed every single control they were shown — stay excluded, holding back 555 votes.

That line is a judgement call and we are showing our work rather than hiding it. Note that the automatic rule alone would have cleared all 31, purely because they had not yet been shown 6 controls each. A rule that needs six trials cannot catch someone who fails their first two.

What This Cannot Tell You

We would rather list these than have a reader find them.

Check It Yourself

The results file is public and signed. It is the same file that draws the leaderboard, so there is no private version.

The file carries its own checksum and signature, when it was generated, and a plain description of the method. Each leaderboard reports how many votes it rests on and how many people were behind them.

Working with us

We run these blind English and Turkish tests on unreleased models and tell you where the pronunciation and rhythm problems are before your users find them. If you build AI voices and want to be in the next round, or want a private evaluation, get in touch through patientdesk.ai. If you think we got something wrong, tell us — we publish corrections next to the original rather than quietly editing it.