I gave the translator two paragraphs: a researcher presenting results, and a line about train tickets costing 24.50 euros. Into Spanish, three times, it came back as two paragraphs with every figure, the name untouched, the URL untouched and the code sample untouched. Into Japanese, three times, the second paragraph was not there at all. Into Telugu, four times, it was missing three times and fused into the first paragraph the fourth. The same input, the same instruction, the same model. The only thing that changed was the target language, and the thing that disappeared was half the text, silently.
TL;DR: This is a small model doing translation, and how well it does depends enormously on how much of that language it was trained on. Measured across ten runs of one 41-word passage: Spanish was correct three times out of three. Japanese dropped the entire second paragraph three times out of three and rendered "14 March" as 14月14日, the fourteenth month. Telugu lost the second paragraph in three runs of four. URLs and code survived all ten. If your target is not a major European language, translate one paragraph at a time and check the output. The translator runs on the site's own server, which is why the honest limits are worth stating.
What was tested, and how
One passage, two paragraphs, forty-one words, carrying the things that usually break in translation: a personal name, a date, a URL, a code sample with quotes and a semicolon, a decimal currency figure, and a sentence whose meaning depends on a specific noun.
Dr. Amara Osei will present the results on 14 March. The full dataset is at
https://example.org/trial-2026/data and the loader expects `read_csv(path, sep=";")`.
Tickets cost 24.50 euros and must be bought before boarding, because the
conductor cannot sell them on the train.
Three target languages, chosen to span what a model like this has seen: Spanish, which is everywhere in training data; Japanese, which is well represented but structurally distant; and Telugu, which has around eighty million speakers and a fraction of the web presence. That gap is the whole subject of the No Language Left Behind work, which exists precisely because translation quality tracks training data rather than speaker count.
Ten runs in total. Here is every one of them.
| Run | Second paragraph | Name left as-is | URL intact | Code intact | Paragraphs |
|---|---|---|---|---|---|
| Spanish 1 | yes | yes | yes | yes | 2 |
| Spanish 2 | yes | yes | yes | yes | 2 |
| Spanish 3 | yes | yes | yes | yes | 2 |
| Japanese 1 | no | no | yes | yes | 1 |
| Japanese 2 | no | no | yes | yes | 1 |
| Japanese 3 | no | no | yes | yes | 1 |
| Telugu 1 | no | no | yes | yes | 1 |
| Telugu 2 | no | no | yes | yes | 1 |
| Telugu 3 | yes | no | yes | yes | 1 |
| Telugu 4 | no | no | yes | yes | 1 |
What happens to half the text?
It is not translated, and nothing says so.
Six of the seven non-Latin runs returned only the first paragraph. The output is fluent, correctly formed and complete-looking. If you do not read Japanese or Telugu, there is no signal at all that the text you pasted was 41 words and the text you received covers 24 of them.
The one Telugu run that kept the second paragraph is not reassuring either. It merged everything into a single paragraph and fused two unrelated clauses, producing a sentence that says the loader must resort to the code because selling on the train is not possible. It also turned the train into a bus.
The cause is visible when you test the dropped paragraph on its own. Sent alone, it translates correctly and keeps the 24.50:
రద్దకెళ్ళకు ముందు టిక్కెట్లు కొనాలి, ఎందుకంటే కమాండర్ వాటిని రైలులో అమ్మలేదు. (24.50 యూరోలు)
So the model can translate that sentence. It stops before reaching it when the input has two paragraphs. That is a generation-length failure rather than a comprehension one, and it is the single most important thing to know about using this tool for a non-European language.
The workaround is a paragraph at a time. Tedious, reliable, and about thirty seconds per paragraph.
What survived every single run?
URLs and code, ten times out of ten.
https://example.org/trial-2026/data intact in all 10
`read_csv(path, sep=";")` intact in all 10
That is the instruction working exactly as intended, and it is not a small thing. A translator that helpfully translates a variable name, localises a decimal separator inside a code sample or rewrites a URL path produces output that is broken in a way nobody notices until something fails. Here the code came back with its quotes, its semicolon and its parameter names untouched in Japanese and Telugu as well as Spanish.
If you are translating documentation, this is the property that matters most, and it is the one that holds.
What happened to the date?
Japanese turned 14 March into the fourteenth month.
14月14日
All three Japanese runs produced it. In Japanese a date is written month-first as 3月14日, one of the per-locale patterns catalogued in Unicode's CLDR, and the model has taken the day number and used it for both fields. There is no fourteenth month, so this is not an ambiguity, it is a wrong date that reads as a plausible one to anyone skimming.
Telugu got it right twice and dropped the month once, returning "the 14th" with no March. Spanish wrote "el 14 de marzo" correctly in all three.
Dates are worth singling out because they are the error most likely to survive a proofread by someone who does not read the target language. A missing paragraph is visible as a length difference. A wrong month is four characters that look like every other date.
Why is the name transliterated when the instruction says to leave it?
Because the instruction is wrong for those scripts, and the model is right.
"Amara Osei" came back unchanged in Spanish and as アマラ・オセ in Japanese and అమరా ఒసే in Telugu. The prompt sent with every request says "Leave names, URLs and code unchanged", and strictly the model disobeyed.
It should. Leaving a Latin-script name in the middle of Japanese prose is not what a Japanese translator does; names are transliterated into katakana as a matter of course, and a document that does not do it reads as machine output. The same applies to Telugu.
So treat that line of the instruction as applying to identifiers rather than to people. If you need a name kept in Latin script, for a passport or a legal document where the spelling is the point, check it and put it back.
Which languages can you rely on?
The twenty offered are not twenty equivalent options, and the difference tracks how much of each language the model saw.
| Tier | Languages | What to expect |
|---|---|---|
| Reliable here | Spanish, French, German, Portuguese, Italian, Dutch | Correct on this test; multi-paragraph input holds |
| Usable with checking | Chinese, Japanese, Korean, Russian, Arabic, Turkish | Structurally fine, but expect truncation and date errors |
| One paragraph at a time | Hindi, Bengali, Telugu, Tamil, Marathi, Kannada, Urdu | Measured truncation; verify every figure |
Only Spanish, Japanese and Telugu were measured. The other rows are placed by how much web text exists in each language, which is the thing that predicts a small model's competence, and they should be treated as expectations rather than findings.
The pattern to carry away: the further down that table you go, the shorter the input should be and the more the output needs checking by someone who reads it.
Is this better or worse than Google Translate?
Worse at translating, better at not sending your document anywhere.
A dedicated translation service runs a much larger model trained specifically on parallel text, and for a language pair like English to Telugu that gap is large. If what you need is the best available translation of a public document, use one of those.
What you get here instead is that the text does not leave the site's own hardware. The things people translate are frequently not publishable: a letter from a lawyer, an email from a landlord, a medical note for a relative, a contract before signing. For those, "good enough to understand, and nobody else saw it" is a different product from "excellent, and logged".
Knowing which of those you need is the whole decision, and it is worth making consciously rather than by whichever tab is open.
Does the tone and formatting survive?
Formatting yes, with the paragraph caveat above. Tone is harder to verify and worth less trust.
The instruction asks for meaning, tone, formatting and line breaks to be kept. In Spanish all three runs preserved the two-paragraph structure exactly. In the non-Latin runs the paragraph break was lost in all seven, including the one that kept both paragraphs' content.
Register is the thing you cannot check without the language. Japanese in particular encodes politeness grammatically, and the difference between a form suitable for a colleague and one suitable for a customer is not something an English speaker can inspect. The runs here used です/ます, the neutral-polite form, which is a reasonable default and may be wrong for your purpose.
If the text is going to a person rather than into your own understanding, have it read by someone who speaks the language. That is true of any machine translation and more true of a small one.
Why does translating it back not prove anything?
Because the round trip is fluent whatever happened in the middle, and it hides exactly the failure you were checking for.
This is the most common way people verify a translation they cannot read: put the output back through, see if English comes out. Here is that test run on the Japanese output above.
Dr. Amal Ose will announce on April 14th. The entire dataset is available at
https://example.org/trial-2026/data. The load arm expects `read_csv(path, sep=";")`.
Read it cold and it looks like a correct translation of a short announcement. Read it beside the original and four things have happened.
The name has decayed through two passes: Amara Osei became アマラ・オセ became Amal Ose. The date has now been wrong twice in different ways, 14 March to the fourteenth month to April 14th, which is a third date that appeared in neither the source nor the Japanese. "The loader" became ロードアーム became "the load arm". And the entire second paragraph is still missing, with nothing in the English to suggest it ever existed.
That last point is the one that matters. A round trip cannot reveal missing content, because the missing content is missing from both ends of it. It tests whether the model can read its own output, which it generally can.
The check that works is the dull one: compare the length of what you pasted against the length of what came back.
Can you translate a whole document?
Not in one go, and the reason is the same failure as above with more at stake.
Very long input is shortened to what the model can read at once, and the page says so when that happens. But the truncation measured here is a different thing, happening at 41 words, well inside any limit: the model simply stops generating before it has finished.
So a document goes through in pieces, and the piece size depends on the target:
| Target | Practical chunk |
|---|---|
| Spanish, French, German, Portuguese, Italian, Dutch | Several paragraphs; check the length |
| Chinese, Japanese, Korean, Russian, Arabic, Turkish | A paragraph or two, checking each |
| Hindi, Bengali, Telugu, Tamil, Marathi, Kannada, Urdu | One paragraph, checking each |
Work in the order the document reads, keep the output in a separate file as you go, and compare the two line counts at the end. For a long document in one of the lower rows that is slow enough to be worth asking whether a dedicated translation service is the better tool, which for public text it usually is.
What chunking does not solve is consistency. A term translated one way in paragraph three may come back differently in paragraph nine, because each request is independent and carries none of the earlier choices. For anything with recurring terminology, settle the key terms first and fix them by hand afterwards.
How long does it take?
Twenty-five to forty seconds for a short passage, and the time tracks the output rather than the input.
| Target | Time | Output |
|---|---|---|
| Spanish | 37 s | 301 bytes |
| Japanese | 25 s | 209 bytes |
| Telugu | 28 s | 335 bytes |
Japanese was fastest because it produced the least, which is the failure described above showing up in the timing. A run that comes back unusually quickly on a non-European target is worth a second look for that reason.
One run in ten returned an upstream 502 rather than a translation. The page handles that correctly, showing an error rather than printing it as output, and a retry worked. Worth knowing it happens on a single shared model.
What should you check before using a translation?
Four things, in the order they went wrong here.
- Length. Compare the output against the input. If it looks short, a paragraph is missing. This catches the worst failure without reading a word of the target language.
- Every number and date. Numbers survive as digits and dates do not survive as dates.
- Code, URLs and identifiers. These held in all ten runs, so this is a quick confirmation rather than a likely problem.
- A human, if it is going to a human. Nothing above substitutes for that, and the politeness register is the reason.
Why does it run on the site's own server?
Because the translation requests people make are a list of things they would rather not upload.
A letter in a language they do not read. A document from a government department. A message from a family member. A clause in a contract. The conventional alternative sends it to a service whose terms allow retention, and for the kinds of text people translate that is a bigger exposure than it is for most tools.
Here it goes to a small model on the site's own hardware and is not stored unless you choose to send feedback. The cost is in the ten rows of the table above, stated plainly rather than discovered.
