The Multilingual Agent Benchmark: What LLM Systems Lose Outside English

Most AI systems are evaluated in English, demonstrated in English, and approved for launch on the strength of English results. Then they are deployed to customers who write in Hindi, Bahasa Indonesia, Arabic, Swahili, or a fluid mix of English and a local language. The published research is consistent on what happens next: accuracy drops, safety guardrails weaken, token costs rise, and the size of each effect depends heavily on the language, the model tier, and the task. This article brings together the strongest public evidence on the multilingual gap, explains why it exists, and sets out a measurement method and a deployment decision framework for teams putting agents into production across languages.

The short answer: the gap is real, uneven, and largest where it is least visible

The headline numbers for frontier models can look reassuring. On OpenAI's Multilingual MMLU (MMMLU), a version of the MMLU test set professionally translated into 14 languages, the strongest reasoning model listed scores between 86% and 91% across most languages¹. But three patterns sit underneath that average, and each matters for production systems.

1. The gap widens sharply for lower-resource languages. The same table shows Yoruba trailing the best-performing language by 13 points for the strongest model and by roughly 30 points for smaller models¹.
2. Smaller, cheaper models lose more. Production agents frequently route high-volume work to smaller models for cost and latency reasons. Those are exactly the models with the largest cross-language spread.
3. Knowledge benchmarks understate the problem for agents. Multiple-choice accuracy does not capture tool-call argument errors, retrieval misses across scripts, or the safety regressions documented in multilingual jailbreak research.

What the published benchmarks show

Knowledge and reasoning: MMMLU

MMMLU is useful because the translations were produced by professional human translators rather than machine translation, which removes one common source of noise in multilingual evaluation¹. The table below extracts five languages across five models from OpenAI's published results. Spanish is used as the high-resource reference because the published table does not include an English row.

Source: OpenAI simple-evals, Multilingual MMLU benchmark results¹. Accuracy figures converted from decimals.

Two observations stand out. First, Hindi and Bengali, languages with hundreds of millions of speakers, sit roughly 1 to 17 points below Spanish depending on the model. Second, the spread for GPT-4o mini (31.6 points between Spanish and Yoruba) is more than double that of o3. A routing decision made for cost reasons can therefore also be a decision about which customers receive a materially less accurate system.

Cultural bias inside "translated" benchmarks

Translation alone does not produce a fair multilingual benchmark. The Global-MMLU study, which covers 42 languages, found that 28% of MMLU questions require culturally sensitive knowledge, and that among questions requiring geographic knowledge, 84.9% focus on North American or European regions². A model can therefore score well on a translated benchmark while still knowing little about the legal, commercial, or cultural context of the users it will serve. The authors release separate culturally sensitive and culturally agnostic subsets so that evaluators can see the difference².

The MILU benchmark makes the same point for India. It spans 11 Indic languages, 8 domains, and 41 subjects, drawing on regional and state-level examinations that cover local history, arts, festivals, and law. Across more than 42 evaluated models, the best performer, GPT-4o, reached 74% average accuracy. Models performed better in high-resource than low-resource languages, and performed poorly in culturally relevant areas such as Arts and Humanities and Law and Governance compared with STEM³. For any agent that answers questions about regulation, benefits, or local procedure, those are precisely the domains that matter.

Multi-step reasoning: MGSM

The MGSM benchmark translated 250 grade-school math problems from GSM8K into ten typologically diverse languages⁴. Its results for PaLM-540B are now dated as absolute numbers, but the structure of the findings remains instructive for agent design.

Source: Shi et al., MGSM, PaLM-540B chain-of-thought results⁴.

The pattern is that pivoting through English recovered a substantial share of the gap: Swahili rose from 35.2% to 51.2%. That is a useful design lever, but it is not free. A translate-first pipeline adds latency, cost, and a new failure point, and it can strip out the very cultural or legal nuance that the Global-MMLU and MILU results show models already struggle with.

Reading comprehension and generation

Belebele, a parallel reading comprehension benchmark covering 122 language variants, found that "despite significant cross-lingual transfer in English-centric LLMs, much smaller MLMs pretrained on balanced multilingual data still understand far more languages," and that larger, deliberately constructed vocabularies correlate with better low-resource performance⁵. For generation, IndicGenBench, which covers 29 Indic languages across 13 scripts and 4 language families, reported "a significant performance gap in all languages compared to English" even for the strongest model tested⁶. The broader MEGA evaluation, spanning 16 datasets and 70 languages, documented the same direction of travel and the particular difficulty of low-resource languages and non-Latin scripts⁷.

Agents and tool use

Agent-specific multilingual evaluation is newer, and its findings are directionally consistent. MAPS translated 805 tasks from GAIA, SWE-bench, MATH, and the Agent Security Benchmark into 11 languages, producing 9,660 language-specific instances. The authors observed degradation in both performance and security when moving from English to other languages, with severity varying by task and correlating with the amount of translated input⁸. A dedicated Arabic tool-use benchmark found that tool-calling accuracy drops by an average of 5 to 10% when users interact in Arabic, regardless of whether tool descriptions are written in Arabic or English⁹.

A 5 to 10% drop in tool-calling accuracy is not a rounding error for an agent that books, refunds, or files. Each failed call is an action taken incorrectly or not at all.

Safety does not transfer evenly

The most consequential multilingual gap is in safety behavior. Yong, Menghini, and Bach showed that translating harmful prompts into low-resource languages bypassed GPT-4's safeguards 79% of the time on AdvBench, comparable to or better than established jailbreak techniques, while high- and mid-resource languages showed substantially lower attack success rates¹⁰. Deng et al. found that low-resource languages exhibited about three times the likelihood of producing harmful content compared with high-resource languages in unintentional scenarios, and that deliberate multilingual attacks produced unsafe output 80.92% of the time for ChatGPT and 40.71% for GPT-4¹¹.

Model providers have improved multilingual safety training since these studies were published, and the specific numbers should not be assumed to hold for current models. The structural lesson does hold: safety evaluations conducted only in English do not establish safety in other languages, and public translation tools make this an attack surface for every deployment, not only multilingual ones.

Why the gap exists

Training data imbalance

Pretraining corpora are dominated by English and a handful of high-resource languages. Belebele's finding that smaller models trained on balanced multilingual data understand more languages than larger English-centric models points directly at data composition rather than model scale as the binding constraint⁵. Safety fine-tuning data is typically even more English-heavy than pretraining data, which helps explain why refusal behavior transfers less reliably than general capability¹¹.

Tokenization

Tokenizers learned mostly from English text split other scripts into more, shorter pieces. Petrov et al. showed that the same text translated into different languages can differ in tokenized length by up to 15 times¹². Ahia et al. found that speakers of many supported languages are "overcharged while obtaining poorer results," and that these speakers tend to come from regions where API access is already less affordable¹³. Longer token sequences also consume context window, which reduces how much retrieved evidence or conversation history an agent can hold.

Tokenizer design is improving. OpenAI's published example sentence for its GPT-4o tokenizer illustrates the scale of the earlier disparity and the size of the correction:

Source: OpenAI, "Hello GPT-4o"¹⁴. Counts are for a single example sentence per language and are illustrative rather than corpus averages.

Under the earlier tokenizer, the Telugu sentence required nearly six times as many tokens as its English equivalent. Even after the improvement it still requires almost twice as many. Token efficiency varies by model family, so per-language cost and context budgets should be measured for the specific model in use rather than assumed.

Evaluation bias

Because most benchmarks originate in English and are then translated, they test whether a model can answer Western-centric questions in another language, not whether it can serve a user in that language's context². Teams that inherit these benchmarks inherit the blind spot.

What this means for production agents

Agents compound the per-step gaps that benchmarks measure in isolation. Four areas deserve specific attention.

• Tool calling and argument extraction. Dates, addresses, personal names, currency amounts, and numerals written in local scripts or local conventions are where structured arguments break. The Arabic tool-use results show this degradation persists even when tool schemas are translated⁹.
• Retrieval. A knowledge base written in English, queried in Tamil, depends on cross-lingual embedding quality and on the model's ability to answer in the user's language from English evidence. Each hop is a place for recall to fall.
• Safety and prompt injection. Multilingual jailbreak research shows guardrails weaken in lower-resource languages¹⁰¹¹, and MAPS found security degradation alongside task degradation in agent settings⁸.
• Code-switching. Real users mix languages within a sentence. Few public benchmarks test this, so it is usually discovered in production rather than before it.

A measurement methodology: per-language scorecards

The published evidence establishes that gaps exist. It cannot tell a team how large the gap is for its own model, prompts, tools, and users. That requires measurement against the actual workload. In our engagements we use the following method; the specific results are always client-specific and are reported per deployment.

1. Build a parallel task set from real traffic

Start from representative production or pilot tasks, not generic benchmarks. Create parallel versions in each target language using native speakers, and add native-origin tasks that have no English equivalent, such as questions about local regulation or regional product variants. This captures both the translation gap and the cultural gap that Global-MMLU and MILU identify.

2. Score each language on the same dimensions

3. Use native-speaker domain reviewers

Automated metrics and LLM-as-judge scoring are themselves weaker outside English, so they cannot be the only check on non-English output. Reviewers should be native speakers who also hold the relevant domain expertise: a clinician fluent in Marathi for a health triage agent, a lawyer fluent in Bahasa Indonesia for a contracts assistant. Reviewers grade against written rubrics, and a sample is double-scored to measure agreement.

4. Report parity ratios, not just scores

Express each language's result as a ratio of the English baseline on the same task set. A parity ratio makes regressions visible when models, prompts, or tokenizers change, and it gives non-technical stakeholders a single figure per language to track.

5. Re-run on every material change

Model upgrades, prompt edits, and new tools can move languages in different directions. The scorecard belongs in the release process, not in a one-time launch review.

A deployment decision framework

With scorecards in hand, each language can be placed into one of four deployment modes. The thresholds below are illustrative starting points that teams should calibrate to the consequence of error in their own domain.

Three questions help place each language:

1. What is the consequence of an error? A wrong answer about store hours and a wrong dosage instruction do not warrant the same threshold.
2. Is the action reversible? Irreversible tool calls such as payments, filings, or deletions in a weaker language should require confirmation.
3. Has safety been tested in this language specifically? If not, the language is not ready, regardless of task accuracy.

Conclusion

The research record is consistent across knowledge benchmarks, reasoning benchmarks, agent benchmarks, and safety studies: performance outside English is lower, less consistent, and less safe, and the gap is largest for lower-resource languages and smaller models. Frontier models have narrowed the gap for major languages, and tokenizer improvements have reduced cost disparities, but neither removes the need to measure. Organizations deploying agents across languages should build per-language scorecards from real tasks, rely on native-speaker domain reviewers rather than English-centric automated judges, and decide language by language whether a system is ready to act on its own.