GSM8K

GSM8K (Grade School Math 8K) is an English-language benchmark of grade-school arithmetic word problems released by OpenAI researchers in 2021. It is used to evaluate whether a large language model can turn a short natural-language problem into a correct numerical answer through several intermediate calculations. The official main configuration contains 8,792 records, although the original paper and repository round the size to 8.5K.^[1][2][3]

GSM8K is primarily a final-answer benchmark. Its reference solutions show intermediate work, but standard automatic scoring extracts and compares the final number. A model can therefore receive credit for the right number even if its written reasoning contains an error, and a correct solution can be marked wrong if the evaluation script fails to extract its answer. Reported scores should be interpreted together with the prompt, decoding method, number of samples, and any calculator or code interpreter used.^[1][2]

Dataset and format

The official distribution has two fields per record: question and answer. The answer is a natural-language solution whose last line places the final numeric result after the delimiter ####. The problems generally require between two and eight steps and use elementary operations such as addition, subtraction, multiplication, and division.^[1][2][3]

Scroll sideways for more →

The dataset card also provides a socratic configuration with the same split sizes. It reformats each solution into a sequence of subquestions and answers rather than changing the underlying problem set.^[3] The repository's main JSON Lines files are the simplest reference distribution. In those files, every archived record has a final integer after ####, while calculation annotations enclosed in << and >> are present in most, but not all, solutions.

GSM8K covers short, self-contained arithmetic stories rather than the full range of mathematical ability. It does not directly test proof construction, advanced algebra, geometry, multilingual understanding, or open-ended mathematical research. Its test split is also small enough that a few corrected labels or extraction failures can visibly change a score.

Creation and quality control

The authors first hired freelance writers through Upwork to produce about 1,000 problems and solutions, then expanded collection through Surge AI. Writers were asked to make problems linguistically varied and to explain their work in natural language. They could use, modify, or ignore seed questions generated by a few-shot-prompted 175-billion-parameter GPT-3 model.^[1][3]

After collection, a different worker re-solved each problem. Cases in which the new final answer disagreed with the original were repaired or removed. In a later check on a smaller subset, 1.7% of problems still produced disagreement between contractors. The authors used this to estimate that fewer than 2% of the original dataset had breaking errors or ambiguities, while explicitly allowing that subtle errors could be more common.^[1][3] This was an agreement-based estimate, not a proof that every remaining question and rationale was correct.

The original paper introduced GSM8K as part of a study on generators and outcome verifiers. A generator produced candidate solutions, and a verifier learned to rank them using labels based on whether the final answer matched the reference. The study sampled 100 candidate solutions per problem for verifier training. It noted that this labeling rule creates false positives when flawed reasoning happens to end at the correct number.^[1]

The paper also defined test@N as the share of problems answered correctly at least once among N separate guesses. This is a coverage measure, not the accuracy of a single selected response, and it is different from majority voting or verifier-ranked accuracy. In one historical ablation, a 6-billion-parameter model trained to emit only the final answer scored 5.2%, compared with 20.6% when trained to generate the full natural-language solution. That result describes one model and training setup; it does not establish a universal gain from displaying intermediate reasoning.^[1]

Evaluation protocols

The usual metric is exact final-answer accuracy on the 1,319-example test split:

accuracy = correctly scored test examples / 1,319

In practice, "correctly scored" depends on normalization and answer extraction. Evaluators commonly remove commas and parse a number from the final response. A reproducible result should identify at least the model version, prompt or demonstrations, decoding temperature, sample count, aggregation rule, answer parser, and any external calculation tool.

GSM8K became a recurring testbed for several chain-of-thought and test-time compute methods.^[4] The following results are historically important but are not a single comparable leaderboard:

Scroll sideways for more →

The self-consistency study averaged its 40-sample evaluation over 10 runs.^[5] The Minerva paper used maj1@k and specified 40 samples for the 540-billion-parameter GSM8K result.^[6] The Program-Aided Language Models result used a code execution environment rather than natural-language arithmetic alone.^[7] These protocol differences are part of the result, so the percentages should not be compared as if only the underlying model changed. Earlier chain-of-thought work likewise showed that prompting with worked reasoning examples could substantially change GSM8K performance without updating model weights.^[4]

Automatic final-answer scoring cannot determine whether a rationale is faithful, logically valid, or copied from a familiar pattern. For research concerned with reasoning quality, GSM8K accuracy is better treated as one outcome measure alongside manual rationale checks, perturbation tests, or process-level evaluation.

Reliability and limitations

Public-test exposure

GSM8K has been publicly available since 2021. Its questions, answers, and many worked model outputs have appeared in repositories, papers, tutorials, and evaluation suites. A high score from a model whose training data are undisclosed cannot by itself separate generalization from direct or indirect data contamination.

GSM1K was created to examine that issue under a controlled pre-release evaluation. Despite its name, the study commissioned 1,205 new problems using human annotators and no language-model-generated data. The authors matched several properties of GSM8K, including human solve rates and solution length. Across the models they tested, the largest GSM8K-to-GSM1K accuracy drop was 8 percentage points, not 13. They also reported a Spearman rank correlation of 0.36 between a model's likelihood of generating GSM8K examples and its performance gap, while noting that many frontier models showed little evidence of overfitting and that all tested models showed broad generalization.^[8]

GSM1K supported evidence of contamination for some evaluated model families, but it did not show that every high GSM8K score was memorized. Its design also cannot permanently guarantee an uncontaminated comparison after the new problems become public.

Labels and final-answer scoring

The original agreement checks found a low rate of obvious disagreement, but later review used different criteria and found additional ambiguity and labeling issues. The GSM8K-Platinum project screened the full 1,319-example test set with multiple models and manually reviewed every example for which at least one model erred. Its published cleaning statistics list 219 flagged examples: 110 rejected as poorly written, 10 relabeled, and 99 verified without a label change. The resulting GSM8K-Platinum set contains 1,209 examples.^[9][10]

Those figures do not directly contradict the original 1.7% disagreement check. The original estimate measured contractor disagreement after the authors' curation process, whereas GSM8K-Platinum used model failures to trigger a later manual review and applied its own criteria for ambiguity and label quality. Results on the 1,209-example revision are therefore not numerically interchangeable with results on the original 1,319-example test set.

Final-answer matching also has a more fundamental limitation: a correct number does not validate the path used to reach it. The original verifier paper and the Minerva study both discuss false positives in which a flawed rationale receives credit because its last answer is correct.^[1][6]

Sensitivity to problem changes

Several derived benchmarks test whether GSM8K performance survives controlled changes:

Scroll sideways for more →

GSM-Symbolic found that changing numerical values could lower accuracy and that performance varied across instantiations of the same template. Increasing the number of clauses generally lowered average performance and increased variance in the tested models. GSM-NoOp produced especially large declines after adding irrelevant but superficially related clauses. Its authors interpreted these findings as evidence of pattern sensitivity and hypothesized that the tested models were not performing genuine formal reasoning; that interpretation is a research conclusion, not something GSM8K alone proves.^[11]

GSM-Plus used GPT-4 to draft question variants and answers, followed by human correction and quality control. It spans eight perturbation types grouped under numerical variation, arithmetic variation, problem understanding, distractor insertion, and critical thinking. The benchmark is useful for paired robustness tests because each variant traces back to a specific GSM8K test question.^[12]

MGSM uses professional human translations and includes typologically diverse languages. It changes the language while largely preserving the mathematical content, so it evaluates a different limitation from GSM-Symbolic and GSM-Plus.^[13]

Access and licensing

OpenAI's grade-school-math repository distributes the original main and socratic data under the MIT License.^[2][3] The official Hugging Face dataset card exposes both configurations and their split metadata.^[3] GSM8K-Platinum keeps the source fields, removes rejected test questions, and adds a cleaning-status field; its additional annotations are licensed under CC BY-SA 4.0, while the underlying GSM8K material remains under its original license.^[10]

For reproducibility, a report should identify whether it used the original GSM8K test split, GSM8K-Platinum, or a derived robustness set. It should also preserve the exact prompt, parser, tool configuration, and sampling procedure rather than reporting only a percentage.

See also

• Model evaluation
• Chain of Thought
• Self-Consistency
• Test-Time Compute
• Data Contamination
• MGSM

References

1. ^Training Verifiers to Solve Math Word Problems - Karl Cobbe et al., arXiv, 2021.
2. ^OpenAI grade-school-math repository - OpenAI.
3. ^Dataset Card for GSM8K - OpenAI on Hugging Face.
4. ^Chain-of-Thought Prompting Elicits Reasoning in Large Language Models - Jason Wei et al., NeurIPS 2022.
5. ^Self-Consistency Improves Chain of Thought Reasoning in Language Models - Xuezhi Wang et al., ICLR 2023.
6. ^Solving Quantitative Reasoning Problems with Language Models - Aitor Lewkowycz et al., NeurIPS 2022.
7. ^PAL: Program-aided Language Models - Luyu Gao et al., ICML 2023.
8. ^A Careful Examination of Large Language Model Performance on Grade School Arithmetic - Hugh Zhang et al., NeurIPS 2024.
9. ^Do Large Language Model Benchmarks Test Reliability? - Joshua Vendrow et al., arXiv, 2025.
10. ^GSM8K-Platinum dataset card - MadryLab on Hugging Face.
11. ^GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models - Seyed Iman Mirzadeh et al., ICLR 2025.
12. ^GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers - Qintong Li et al., ACL 2024.
13. ^Language Models are Multilingual Chain-of-Thought Reasoners - Freda Shi et al., arXiv, 2022.