I'm a Slovenian developer and I talk to Claude Code in Slovenian. About 2.5 million people speak my language. Every prompt engineering guide is written in English, and the usual advice is: prompt in English, the models are simply better at it. I wanted to know if that's actually true for daily coding work, so I measured it. Here's what I found.
How I tested it
Three parts:
- My real usage. I parsed all my Claude Code session transcripts from two environments (WSL for personal projects, Windows PowerShell for my day job): 2,319 prompts by my first count, 1,899 after the strictest filtering pass that drops pasted logs, tool output and machine-generated turns.
- Published research. Multilingual benchmarks, interpretability studies, tokenizer papers, official vendor docs.
- My own token measurements. Real prompts of mine, translated to English, counted with different tokenizers.
Then one more thing: I gave the exact same research task to OpenAI Codex (GPT-5.6), isolated from my results, and compared what the two systems concluded. More on that below.
What my usage actually looks like
| WSL | Windows | |
|---|---|---|
| Real prompts (strict filter) | 1,362 | 537 |
| Slovenian, where language is determinable | 98.7% | 99.4% |
| Median prompt length | 54 chars | 90 chars |
So I don't "mostly" prompt in Slovenian. I prompt in Slovenian, full stop. Fast, with typos, often without diacritics (š, č, ž), with English technical terms mixed in wherever they're natural. The question is whether any of that hurts.
What the research says about quality
Short version: it hurts less than I expected, if your model is big enough.
Models internally lean on English anyway. Interpretability work from EPFL (Do Llamas Work in English?) and Anthropic's own circuit-tracing study (On the Biology of a Large Language Model) points the same way: large models process meaning in a shared concept space that sits closest to English. Non-English input adds a thin translation layer at the edges. The thinking in the middle is mostly language-neutral. This also means instructions like "think in English, respond in Slovenian" are mostly wasted words on modern models.
The gap was real, and it has mostly closed at the frontier. In 2022, PaLM solved 62% of math word problems in English but only 47% averaged across underrepresented languages (MGSM). Today, Anthropic's official numbers for benchmarked languages show Spanish at 98.2% of English performance and German at 97.0% (multilingual support docs). No vendor benchmarks Slovenian, but a Slovenian pragmatics benchmark (SloPragEval) measured GPT-5 at 0.81 in Slovenian vs 0.83 in English. Two points. Not nothing, but not a reason to switch languages for daily work.
Small models are a different story. The same Anthropic table shows Claude Opus 4.1 keeping 80% of its English performance in Yoruba, while the much smaller Haiku 4.5 keeps only 53%. If a small cheap model handles your non-English workload, test it in your language before trusting it.
Mixing languages is fine. In my direction, it even helps. A 2025 study (Lost in the Mix) found an asymmetry: foreign words inserted into English text hurt comprehension, but English words inserted into non-English text often improve it, by up to 13 points on some models. Slovenian sentences with English technical terms left as-is turn out to be the good kind of mixing.
What Slovenian costs in tokens
The literature says non-English text is expensive: Slovenian pays about 1.88x the tokens of English on GPT-4's old cl100k tokenizer (Petrov et al., NeurIPS 2023). That number worried me. Then I measured my own prompts: 30 real ones, each translated to English faithfully.
| Tokenizer | Slovenian premium on my real prompts |
|---|---|
| cl100k_base (GPT-4 era) | +34% |
| o200k_base (GPT-4o / GPT-5) | +16% |
Why so much lower than the paper's numbers? Because real developer prompts aren't clean prose. They're full of file paths, SQL, error messages and English tech terms that tokenize identically in both languages.
Two results I didn't expect:
- Diacritics are free. The same text with and without š/č/ž differs by 0.6% in tokens. My lazy typing habit costs nothing.
- The cost that matters is on the output side. Output tokens are about 5x the price of input tokens and they dominate latency. A short Slovenian question costs nothing worth mentioning. A long Slovenian document generated by the model costs 16-34% more time and money than the same document in English.
One honest caveat: Claude's tokenizer is not public, so these are OpenAI tokenizer numbers (plus Gemini's count API, which agreed). Published cross-vendor comparisons suggest Claude's non-English premium is somewhat higher.
The twist: I had Codex repeat the whole study
To check my own work, I gave the identical research brief to OpenAI Codex (GPT-5.6): same task wording, same environment facts, an isolated working directory, and no access to my report. It even had to exclude the transcript of the session that ordered the research, so it couldn't peek at my conclusions.
It came back with the same core answer: keep prompting in Slovenian, keep technical identifiers verbatim, write reusable instructions in English with native trigger phrases, don't translate domain knowledge. Two systems independently converging on the same conclusions is worth more than either report alone.
And Codex found one thing I had missed. It manually reviewed 50 of my prompts and rated 90% of them as clearly understandable despite all the typos. The weak ones weren't weak because they were Slovenian. They were weak because they were vague: "fix this", "now do it properly", no success criteria, no constraints. Its conclusion, which I'm keeping: the biggest improvement isn't the language, it's explicitness.
What I changed, and what I kept
Kept:
- Slovenian for interactive work. I think faster and state intent more precisely in my own language.
- English technical terms inside Slovenian sentences. The research says this direction of mixing is harmless to helpful.
- Everything technical verbatim. Package names, SQL, paths, error messages, config keys. Never translated, never paraphrased.
- Hybrid skills. Reusable agent instructions in English, trigger phrases in the description in Slovenian (the words I actually type), business domain knowledge in Slovenian because the terminology lives in Slovenian.
Changed:
- An explicit language rule in my global config, because Claude Code is known to drift back to English after context compaction:
Always respond in Slovenian unless explicitly asked otherwise.
Keep code, identifiers, commands, paths and error messages
in their original form.
Enter fullscreen mode Exit fullscreen mode
A five-field template for bigger tasks: goal, context, constraints, verification, output format. This fixes more than any language switch would.
English selectively: long structured specs and agent briefs, or an A/B retry when a hard reasoning task gives a shaky answer.
Wrapping up
For frontier models, the language you prompt in matters far less than how explicit your prompt is. Prompt in the language you think fastest in, keep everything technical verbatim, write reusable instructions in English with trigger phrases in your own language, and spend the effort you save on clear goals and constraints instead of translation.
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.