MGSM (Multilingual Grade School Math)

MGSM
Overview
Full name	Multilingual Grade School Math
Abbreviation	MGSM
Description	A multilingual benchmark evaluating mathematical reasoning across 10 typologically diverse languages using grade-school math problems
Release date	2022-10-06
Latest version	1.0 (original); Rev2 (corrected, 2025)
Benchmark updated	2025 (MGSM-Rev2)
Authors	Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei
Organization	Google Research
Technical Details
Type	Mathematical Reasoning, Multilingual Evaluation
Modality	Text
Task format	Word problems requiring multi-step arithmetic
Number of tasks	10 languages (plus English source)
Total examples	2,750 (250 per language, 11 languages including English)
Evaluation metric	Exact match accuracy
Domains	Elementary mathematics, Arithmetic word problems
Languages	English, Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, Telugu
Performance
Human performance	Based on GSM8K validation
Baseline	Varies by language and model
SOTA score	~96% (frontier models, 2025)
SOTA model	GPT-5.3 Codex, Claude Opus 4.5 (Thinking)
SOTA date	2025
Saturated	Yes (top models exceed 95%)
Resources
Website	Official website
Paper	Paper (arXiv)
Venue	ICLR 2023
GitHub	Repository
Dataset	Download (Hugging Face)
Corrected version	MGSM-Rev2
License	CC-BY-SA 4.0
Predecessor	GSM8K

MGSM (Multilingual Grade School Math) is a multilingual benchmark that measures the mathematical reasoning ability of large language models by taking 250 grade-school math word problems from GSM8K and human-translating them into 10 typologically diverse languages. It was introduced by Google Research in the paper "Language Models are Multilingual Chain-of-Thought Reasoners" (Shi et al., October 2022), published at ICLR 2023, and is the standard test for whether chain-of-thought reasoning transfers across languages^[1]. The paper's central finding is that mathematical reasoning emerges as a near-universal capability in sufficiently large models, holding up even in underrepresented languages such as Bengali and Swahili that make up less than 0.01% of typical pretraining corpora^[1].

As of 2025, MGSM is one of the most widely used multilingual reasoning benchmarks. It is integrated into major evaluation frameworks such as EleutherAI's Language Model Evaluation Harness and appears on leaderboards maintained by Kaggle, Vals AI, and others. Frontier models now routinely exceed 90% average accuracy across all languages, pushing the benchmark toward saturation and prompting harder successors such as MGSM-Rev2 and MGSM-Pro^[6][15].

What is MGSM?

MGSM, short for Multilingual Grade School Math, is a fixed set of 250 elementary arithmetic word problems sampled from the GSM8K test split and manually translated, by professional native speakers, into 10 languages plus the English source, for a total of 2,750 problems across 11 language versions^[1]. Because the problem content is held constant and only the language varies, MGSM isolates the effect of language on a model's reasoning, rather than conflating language with problem difficulty. The benchmark's authors describe its construction plainly in the opening of the paper: "We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages."^[1]

Background and motivation

Before MGSM, most evaluations of mathematical reasoning in language models were conducted exclusively in English. The GSM8K dataset, introduced by Cobbe et al. at OpenAI in 2021, provided 8,500 linguistically diverse grade-school math word problems and quickly became a standard benchmark for testing arithmetic reasoning^[2]. However, GSM8K told researchers nothing about whether the reasoning abilities it measured would generalize to other languages.

This gap mattered for several reasons. First, the majority of the world's population does not speak English as a first language, so practical deployment of reasoning-capable models requires multilingual competence. Second, researchers wanted to understand whether chain-of-thought reasoning, the prompting technique introduced by Wei et al. in 2022 that dramatically improved math performance in English, would work across languages with different syntactic structures, writing systems, and levels of representation in training data^[3]. Third, the field lacked a controlled experimental setup for measuring cross-lingual reasoning transfer, since existing multilingual benchmarks focused on tasks like natural language inference and question answering rather than multi-step mathematical problem-solving.

Shi et al. designed MGSM to fill this gap. By taking a fixed set of 250 problems from GSM8K's test split and translating them into 10 carefully selected languages, the researchers created a controlled environment where problem content remained constant and only the language varied. This design made it possible to isolate the effect of language on reasoning performance^[1].

How was the MGSM dataset built?

Source problems

The 250 source problems were drawn from the GSM8K test set. Each problem is an elementary-level arithmetic word problem requiring between 2 and 8 sequential reasoning steps. The problems involve basic operations (addition, subtraction, multiplication, and division) and produce integer answers. A typical problem describes a real-world scenario, such as buying fruit at a market or dividing objects among people, and asks for a single numerical result. The GSM8K dataset itself was written by human problem authors with comprehensive quality control, achieving an estimated error rate below 2%^[2].

Which languages does MGSM cover?

The 10 target languages were chosen to maximize typological diversity while spanning a range of resource levels in LLM pretraining data. They cover eight language families, multiple writing systems, and both high-resource and low-resource settings.

Scroll sideways for more →

The selection intentionally included languages at the extremes of pretraining data availability. English, the dominant language in most LLM training sets, sits at one end, while Telugu, which constitutes roughly 0.0002% of PaLM's pretraining corpus, sits at the other^[1][4]. This range allowed the researchers to study how pretraining data volume affects multilingual reasoning.

How were the problems translated?

The translation methodology prioritized accuracy and naturalness. Between 1 and 5 professional native-speaker translators worked on each language. Every translator had at least 2 years of professional experience and was contractually prohibited from using machine translation^[1][5]. The process included several quality-control steps:

1. Initial translation: A native speaker translated each problem and its solution into the target language.
2. Cultural adaptation: Where necessary, translators adapted names, units, and contextual references to be natural in the target language while preserving the mathematical structure.
3. Verification: At least one additional reviewer checked each translation for mathematical equivalence and linguistic naturalness.
4. Consistency checks: The research team verified that answer values remained identical across all language versions.

Despite these precautions, later research revealed that some translation errors survived the review process, leading to the development of MGSM-Rev2 (discussed in a later section)^[6].

How is MGSM evaluated and scored?

Prompting strategies

Shi et al. tested four distinct prompting strategies on MGSM, each designed to probe a different aspect of multilingual reasoning:

Scroll sideways for more →

All experiments used 8-shot prompting, meaning 8 exemplar problem-solution pairs were provided in the prompt before the test question. The exemplars were drawn from the GSM8K training set, and for the NATIVE-COT and DIRECT strategies, the exemplars were translated into the corresponding target language^[1].

How is MGSM scored?

MGSM uses exact match on the final numerical answer. The model's output is parsed for a number following a standardized "The answer is" format, and this number is compared to the gold answer. The evaluation uses greedy decoding (temperature 0) to ensure reproducibility. No partial credit is given for correct intermediate steps with a wrong final answer^[1].

Self-consistency

In addition to standard greedy decoding, the researchers evaluated self-consistency (SC), a technique introduced by Wang et al. (2022) in which multiple reasoning chains are sampled and the most common final answer is selected by majority vote^[7]. Self-consistency produced consistent improvements across all languages, with particularly large gains on low-resource languages.

What did the original MGSM experiments find?

Models tested

The original paper evaluated three model families:

Scroll sideways for more →

Key results from PaLM-540B

PaLM-540B with the TRANSLATE-EN strategy achieved the best overall performance, with an average accuracy of 55.0% across all languages. The EN-COT strategy followed at 51.3% average accuracy, and NATIVE-COT achieved 48.1%. The DIRECT strategy, which provided no intermediate reasoning steps, performed substantially worse at under 20% on most languages^[1][8].

Selected per-language results for PaLM-540B (TRANSLATE-EN strategy)^[1]:

Scroll sideways for more →

These results were surprising because Bengali and Swahili, which each account for less than 0.01% of PaLM's pretraining data, still achieved over 50% accuracy. The gap between the best-performing language (English at 62.4%) and the worst-performing languages was roughly 10 to 12 percentage points with the TRANSLATE-EN approach^[1].

Flan-PaLM results

The paper also reported results for Flan-PaLM, the instruction-finetuned version of PaLM. Flan-PaLM with chain-of-thought prompting and self-consistency achieved substantially higher scores, including 69.6% on Bengali, demonstrating that instruction tuning can meaningfully improve multilingual reasoning^[9].

Does multilingual reasoning depend on model scale?

One of the paper's central findings was that multilingual reasoning ability is strongly scale-dependent. At 8 billion parameters, PaLM showed minimal ability to solve MGSM problems in any language, with chain-of-thought prompting providing little benefit. At 62 billion parameters, English performance improved meaningfully, but non-English languages lagged behind. At 540 billion parameters, reasoning ability emerged across all tested languages, including low-resource ones^[1].

This pattern matched the broader finding from Wei et al. (2022) that chain-of-thought reasoning is an emergent capability that appears at sufficient model scale^[3]. MGSM extended this observation to the multilingual setting, showing that the scale threshold for non-English reasoning is roughly similar to that for English reasoning.

Impact of chain-of-thought prompting

Across all languages and model sizes, chain-of-thought prompting produced large improvements over direct answer prediction:

Scroll sideways for more →

The results showed that English chain-of-thought (EN-COT) slightly outperformed native-language chain-of-thought (NATIVE-COT) for most languages. This suggests that models have stronger reasoning capabilities in English, their dominant training language, and can leverage English reasoning even when the question is posed in another language^[1].

How does MGSM performance vary across languages?

The language gap

The original MGSM results showed a notable performance gap between English and other languages, particularly low-resource ones. This "language gap" was initially reported as ranging from 15 to 40 percentage points depending on the model and prompting strategy^[1]. For instance, with the NATIVE-COT strategy on PaLM-540B, the gap between English and the lowest-performing language could exceed 20 percentage points.

However, as discussed in the section on MGSM-Rev2 below, subsequent research showed that a substantial portion of this gap was artifactual, caused by translation errors and inconsistent answer-extraction scripts rather than genuine differences in model capability^[6].

Resource level and performance

Despite the caveats about translation quality, a real correlation between pretraining data volume and benchmark performance does exist. High-resource languages (English, Spanish, French, German) consistently score higher than medium-resource languages (Chinese, Japanese, Thai), which in turn score higher than low-resource languages (Bengali, Swahili, Telugu). The magnitude of this gap decreases with model scale, however. In the largest models, even low-resource languages achieve competitive performance^[1].

Error pattern analysis

Analysis of model errors across languages revealed several patterns:

Scroll sideways for more →

Arithmetic errors occurred at similar rates regardless of language, supporting the idea that basic computation is language-agnostic. However, errors in problem comprehension and step organization were more frequent in low-resource languages, suggesting that the language modeling component (as opposed to the arithmetic component) is the primary bottleneck in multilingual math reasoning^[1].

What is MGSM-Rev2?

MGSM-Rev2 is a corrected 2025 release of MGSM that fixes mistranslations and standardizes answer extraction, after researchers found that much of the benchmark's apparent "language gap" was an artifact of evaluation flaws rather than model ability^[6][10].

Discovery of errors

In 2025, Peter et al. published "Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results," which systematically examined translation quality in the original MGSM dataset^[6]. The study uncovered several categories of errors:

Semantic distortions: In the German translation, the English phrase "An orange costs 5 less than what a watermelon cost" was rendered as "Eine Orange kostet 5 Mal weniger" ("5 times less"), fundamentally changing the mathematical relationship in the problem^[6].

Logic inversions: One German translation changed "How many girls are not in the girl scout?" to "Wie viele Madchen sind bei den Pfadfinderinnen?" ("How many girls are in the scout?"), inverting the question entirely^[6].

Constraint changes: "Round all his prices to the nearest dollar" was translated into German as rounding up specifically, adding a constraint not present in the original English^[6].

Temporal errors: In at least one case, "Tuesday" was mistranslated as "Thursday" in the German version^[6].

Answer extraction issues

Beyond translation errors, Peter et al. found that inconsistent answer-extraction scripts compounded the problem. The original evaluation code used language-specific answer prefixes and assumed English number formatting (periods for decimals, commas for thousands separators). For languages like French, where number formatting conventions differ, this caused correct answers to be scored as incorrect. Bengali numerals posed a particular challenge; when Bengali native numerals were not converted to Arabic equivalents, accuracy dropped dramatically^[6].

Improving answer extraction alone yielded a 10-percentage-point boost for French on GPT-5^[6].

Corrections and the revised dataset

The corrections fell into three main categories:

1. Clarity improvements: Resolving ambiguous phrasing (the most frequent correction type)
2. Numerical and factual fixes: Correcting incorrect values or logical relationships
3. Unit specification: Explicitly stating required answer units

Google released MGSM-Rev2 on GitHub, incorporating all identified corrections. Two erroneous English source questions were corrected, ambiguous questions were rephrased, and all translations were redone using Gemini with subsequent verification to ensure every question was answerable^[10].

Impact on the language gap

The corrections had a dramatic effect on reported cross-lingual performance gaps:

Scroll sideways for more →

For Gemma 3 27B, the Bengali accuracy jumped from 45.2% to 91.2% after correcting numeral handling, a 46-percentage-point improvement that was entirely an artifact of evaluation methodology rather than model capability^[6].

After combining corrected translations with improved answer extraction, the maximum cross-lingual accuracy gap for strong models shrank to under 2 percentage points for Gemini 2.5 Pro, under 1.6% for GPT-5, and under 2% for Claude Sonnet. The authors concluded that the language gap "mostly disappears" once these fixes are applied, "leading to completely different conclusions" from those in the original paper^[6].

What is MGSM-Pro?

MGSM-Pro is a 2026 robustness extension of MGSM that generates at least five variants of each problem by changing names, digits, and adding irrelevant context, so that scores reflect genuine reasoning rather than memorization of the 250 fixed problems^[15].

Motivation

Even with MGSM-Rev2 correcting translation errors, researchers identified another limitation: models might memorize the specific numerical values in MGSM's 250 problems, since the benchmark has been publicly available since 2022 and is widely used in training evaluations. To test this concern, Xu et al. (2026) introduced MGSM-Pro, which applies the GSM-Symbolic approach of varying surface-level details while preserving mathematical structure^[15].

Methodology

MGSM-Pro creates multiple instantiations of each MGSM problem through two series of modifications:

Scroll sideways for more →

The dataset covers nine languages: four high-resource (English, Chinese, French, Japanese) and five low-resource (Swahili, Amharic, Igbo, Yoruba, Twi). Templates were created for 225 of the 250 MGSM questions, translated using Gemini 2.0 Flash, and verified by native speakers^[15].

Key findings

Changing entity names had minimal impact on performance. However, changing numerical values caused large accuracy drops, particularly for low-resource languages. The IC_# configuration (irrelevant context plus changed numbers) produced the most severe degradation^[15].

Robustness varied significantly across models:

Scroll sideways for more →

Claude Sonnet 4 proved the most robust, moving from 2nd place on the original MGSM to 1st place on MGSM-Pro. Gemini 2.5 Flash, despite having the highest original accuracy, dropped to 4th place, indicating that strong performance on static benchmarks does not guarantee robustness to surface-level variations^[15].

Low-resource languages suffered the most severe degradation. For Gemma 3 27B, Twi accuracy dropped from 19.6% to 9.4% and Yoruba from 45.3% to 31.2% under the IC_# configuration^[15]. The authors recommended evaluating each problem using at least five digit-varying instantiations to obtain reliable accuracy estimates.

What is the current MGSM leaderboard, and is the benchmark saturated?

Model performance (2025)

As of early 2025, the MGSM leaderboard (based on the original dataset) shows the following top performers:

Scroll sideways for more →

Frontier reasoning models such as Claude Opus 4.5 (Thinking) have reached 95.2% on MGSM, and GPT-5.3 Codex has been reported at 96%^[11].

Is MGSM saturated?

The benchmark is now considered saturated for frontier models. When the top 10 models all exceed 90% average accuracy and the best exceed 95%, the benchmark loses its ability to distinguish between models' multilingual reasoning capabilities. Vals AI, which maintains a model evaluation platform, has stopped running MGSM on new model releases for this reason^[12]. The benchmark remains useful for evaluating smaller or open-source models, where a wider performance spread still exists. Models below 10 billion parameters, for example, typically score under 70% on MGSM^[13].

Where is MGSM used?

Evaluation frameworks

MGSM is integrated into several widely used evaluation tools:

Scroll sideways for more →

The EleutherAI evaluation harness supports both mgsm_direct and mgsm_cot_native task variants for all 11 languages, with configuration files specifying few-shot counts, answer extraction patterns, and scoring logic^[14]. The harness is used internally by organizations including NVIDIA, Cohere, BigScience, and Mosaic ML.

Use in model releases

MGSM scores frequently appear in technical reports for major model releases. Google's PaLM 2 technical report used MGSM to demonstrate multilingual improvements over the original PaLM^[4]. Meta's Llama 3 series reports include MGSM results. Anthropic, OpenAI, and Google DeepMind all reference MGSM when reporting multilingual capabilities of their respective model families.

Downstream research

MGSM has spawned several lines of follow-up research:

MGSM8KInstruct: An extension that automatically translated all 7,473 GSM8K training problems into 10 languages, producing training data for multilingual math instruction tuning. The MathOctopus family of models, fine-tuned on MGSM8KInstruct, demonstrated that cross-lingual fine-tuning (training on English questions with target-language reasoning) can boost even monolingual English performance; MathOctopus-7B gained 8.4 percentage points on the English GSM8K through multilingual training^[16].

mCoT (Multilingual Chain-of-Thought): Research on multilingual instruction tuning for reasoning consistency, using MGSM as the primary evaluation benchmark^[17].

Zero-shot multilingual CoT: Studies on improving zero-shot chain-of-thought reasoning across languages, building on the MGSM findings that few-shot prompting is not strictly necessary for multilingual reasoning in large enough models^[18].

How does MGSM relate to GSM8K?

MGSM is a direct subset and multilingual extension of GSM8K. The relationship between the two benchmarks is straightforward:

Scroll sideways for more →

The 250 MGSM problems are a proper subset of the 1,000 GSM8K test problems. Any model's MGSM English score should therefore be comparable to (though not identical to) its GSM8K score, since MGSM uses a different subset and different few-shot exemplars^[1][2].

Related benchmarks

Scroll sideways for more →

What are the limitations of MGSM?

Dataset size

With only 250 problems per language, MGSM is a relatively small benchmark. Statistical noise from this small sample size can affect conclusions, particularly when comparing models with similar performance levels. A 2-percentage-point difference (5 problems) may not be statistically significant^[1].

Problem difficulty

The problems are elementary-level, requiring only basic arithmetic. This means MGSM does not test higher mathematical reasoning such as algebra, geometry, probability, or calculus. Models that struggle with MGSM problems face fundamental limitations, but models that ace MGSM may still fail on more advanced mathematics^[1].

Translation artifacts

Even with the Rev2 corrections, machine-translated or human-translated math problems may contain subtle cultural or linguistic artifacts that affect difficulty in ways unrelated to mathematical reasoning. Problem contexts (shopping scenarios, school situations) may feel more or less natural depending on the cultural background of the target language^[6].

Contamination risk

Because MGSM has been publicly available since October 2022 and is included in many benchmark suites, there is a risk of data contamination, where models may have seen MGSM problems during pretraining or fine-tuning. MGSM-Pro partially addresses this through numerical variation, but the original 250 problems remain fixed^[15].

Limited language coverage

Ten languages, while typologically diverse, represent a small fraction of the world's approximately 7,000 languages. Notable omissions include Arabic, Hindi, Korean, Indonesian, and virtually all indigenous and minority languages. Efforts like BenchMAX are beginning to address this gap^[19].

See also

• GSM8K
• Chain-of-thought prompting
• Benchmark
• PaLM
• Multilingual AI
• Mathematical reasoning
• BIG-Bench Hard
• MMLU

References

1. Shi, F., Suzgun, M., Freitag, M., Wang, X., Srivats, S., Vosoughi, S., Chung, H. W., Tay, Y., Ruder, S., Zhou, D., Das, D., & Wei, J. (2022). "Language Models are Multilingual Chain-of-Thought Reasoners." arXiv:2210.03057. Published at ICLR 2023. https://arxiv.org/abs/2210.03057 ↩
2. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). "Training Verifiers to Solve Math Word Problems." arXiv:2110.14168. https://arxiv.org/abs/2110.14168 ↩
3. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 2022. https://arxiv.org/abs/2201.11903 ↩
4. Google. (2023). "PaLM 2 Technical Report." https://ai.google/static/documents/palm2techreport.pdf ↩
5. Emergent Mind. "MGSM Benchmark: Multilingual Grade School Math." https://www.emergentmind.com/topics/multilingual-grade-school-math-mgsm-benchmark ↩
6. Peter, J.-T., Vilar, D., Domhan, T., Malkin, D., & Freitag, M. (2025). "Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results." arXiv:2511.05162. https://arxiv.org/abs/2511.05162 ↩
7. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." arXiv:2203.11171. https://arxiv.org/abs/2203.11171 ↩
8. Liner. "Quick Review: Language Models are Multilingual Chain-of-Thought Reasoners." https://liner.com/review/language-models-are-multilingual-chainofthought-reasoners ↩
9. Chung, H. W., Hou, L., Longpre, S., et al. (2022). "Scaling Instruction-Finetuned Language Models." arXiv:2210.11416. https://arxiv.org/abs/2210.11416 ↩
10. Google Research Datasets. "MGSM-Rev2." GitHub. https://github.com/google-research-datasets/MGSM-Rev2 ↩
11. llm-stats.com. "MGSM Benchmark Leaderboard." https://llm-stats.com/benchmarks/mgsm ↩
12. Vals AI. "MGSM Benchmark." https://www.vals.ai/benchmarks/mgsm ↩
13. llmdb.com. "MGSM - LLM Benchmark." https://llmdb.com/benchmarks/mgsm ↩
14. EleutherAI. "lm-evaluation-harness: MGSM task." GitHub. https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mgsm/README.md ↩
15. Xu, T. et al. (2026). "MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation." arXiv:2601.21225. https://arxiv.org/abs/2601.21225 ↩
16. Chen, X. et al. (2024). "Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations." JMLR 25. https://jmlr.org/papers/volume25/23-0870/23-0870.pdf ↩
17. Qin, L. et al. (2024). "mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models." ACL 2024. https://aclanthology.org/2024.acl-long.649.pdf ↩
18. Huang, Y. et al. (2023). "Improving Zero-shot Chain-of-Thought Reasoning across Languages." EMNLP 2023. https://aclanthology.org/2023.emnlp-main.163.pdf ↩
19. Li, Y. et al. (2025). "BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models." arXiv:2502.07346. https://arxiv.org/abs/2502.07346 ↩