# COPA (Choice of Plausible Alternatives)

> Source: https://aiwiki.ai/wiki/copa
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: AI Benchmarks, Data & Datasets, Natural Language Processing
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "COPA (Choice of Plausible Alternatives)." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/copa
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**COPA (Choice of Plausible Alternatives)** is an English-language [benchmark](https://aiwiki.ai/wiki/benchmark) of 1,000 two-choice questions that tests open-domain commonsense causal reasoning: given a short premise, a system must pick which of two alternatives is the more plausible cause or effect of it.[1][2] Melissa Roemmele, Cosmin Adrian Bejan and Andrew S. Gordon introduced it at the AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, held at Stanford University on March 21-23, 2011.[1][2] The questions are split evenly into a development set and a test set of 500 each, and the position of the correct answer is randomized, so random guessing scores 50%.[1][2]

COPA was hard for a long time. The statistical systems of the early 2010s scored in the 58-65% range on the test set while human raters agreed almost perfectly.[1][3] It became the shared task of SemEval-2012 Task 7, then one of the eight tasks of the [SuperGLUE](https://aiwiki.ai/wiki/superglue) benchmark in 2019, where human performance was estimated at 100%.[3][4] Pretrained [transformer](https://aiwiki.ai/wiki/transformers) models and later [large language models](https://aiwiki.ai/wiki/large_language_model) closed the gap: fine-tuned models passed 98% on the SuperGLUE test set by 2021 and 2022, and in 2024 the benchmark's own authors described COPA as "no longer as challenging as it previously was."[8][9][10] COPA is part of the wider family of [commonsense reasoning](https://aiwiki.ai/wiki/commonsense_reasoning) benchmarks that also includes the [Winograd Schema Challenge](https://aiwiki.ai/wiki/winograd_schema_challenge), [HellaSwag](https://aiwiki.ai/wiki/hellaswag), [PIQA](https://aiwiki.ai/wiki/piqa) and [WinoGrande](https://aiwiki.ai/wiki/winogrande).

## Overview

| Attribute | Value |
|---|---|
| Full name | Choice Of Plausible Alternatives |
| Authors | Melissa Roemmele, Cosmin Adrian Bejan, Andrew S. Gordon |
| Institutions | Indiana University (Department of Linguistics); University of Southern California, Institute for Creative Technologies |
| Introduced | AAAI Spring Symposium, Stanford University, March 2011 |
| Size | 1,000 questions (500 development, 500 test) |
| Format | Premise plus two alternatives; choose the more plausible cause or effect |
| Metric | Accuracy (random baseline 50%) |
| Human agreement | Cohen's kappa 0.965 between two raters per question |
| License | BSD 2-Clause (copyright 2010, University of Southern California) |
| Later inclusion | SemEval-2012 Task 7; SuperGLUE (2019) |

Sources: [1][2][4]

## Background and motivation

Before COPA, logical formalizations of commonsense knowledge were usually evaluated with challenge problems of varying complexity, an approach associated with John McCarthy's notion of elaboration tolerance and with Leora Morgenstern's Common Sense Problem Page.[1] Roemmele, Bejan and Gordon listed two drawbacks of that approach. Whether a system had solved a problem was a subjective judgment that typically took a full paper to argue, and the focus on a few deep puzzles favored depth over breadth, a particular problem for research programs such as [Cyc](https://aiwiki.ai/wiki/cyc) and ConceptNet whose explicit goal was broad inferential coverage.[1]

The authors took their model instead from [natural language processing](https://aiwiki.ai/wiki/natural_language_processing), specifically the Recognizing Textual Entailment (RTE) challenges run by the PASCAL Network of Excellence from 2004 to 2007 and by NIST from 2008.[1] RTE question sets had well-defined inputs and outputs, answers validated by several human raters, enough questions to reward breadth, separate development and test sets, and a 50/50 balance that made gains over chance easy to see.[1] RTE, however, asked about entailment: inferences that are necessarily true given the meaning of a text. Commonsense reasoning is mostly about implication, the likely causes and effects of an event, which are judged in degrees of plausibility rather than as strictly true or false.[1] COPA kept the RTE evaluation design but changed the question to one of relative causal plausibility.[1]

The paper grounds this choice in theories of causality from philosophy and psychology. It discusses the "necessity in the circumstances" and "sufficiency in the circumstances" criteria, mechanism views of causal reasoning, and causal bridging inferences, in which a reader supplies a piece of background knowledge (for example, that balloons rise) linking two events.[1] It then adopts a cost-based view of plausibility: the correct alternative is the one whose bridging inferences are cheaper to add to the "causal field" of background assumptions.[1]

## Task format

Each COPA question has three parts: a premise, a question that asks either for a cause or for a result, and two alternatives.[1][2] Both alternatives are meant to be conceivable; the task is to choose the one that more plausibly stands in a causal relation to the premise.[1][3] Questions come in two directions:

- **Forward causal reasoning** asks what happened as a result of the premise.
- **Backward causal reasoning** asks what caused the premise.

The original paper and the SemEval task description use these two examples:[1][3]

| Direction | Premise | Alternative 1 | Alternative 2 | Answer |
|---|---|---|---|---|
| Forward (result) | The man lost his balance on the ladder. | He fell off the ladder. | He climbed up the ladder. | 1 |
| Backward (cause) | The man fell unconscious. | The assailant struck the man in the head. | The assailant took the man's wallet. | 1 |

The COPA homepage gives further examples, such as "The man broke his toe. What was the CAUSE of this?" with the alternatives "He got a hole in his sock" and "He dropped a hammer on his foot," and "I tipped the bottle. What happened as a RESULT?" with "The liquid in the bottle froze" and "The liquid in the bottle poured out."[2]

The wrong alternative is the hard part to write. In the assault example, taking the man's wallet fits the same scenario as striking him, but it is a less plausible direct cause of his falling unconscious. The authors wrote wrong alternatives this way so that the questions would require causal reasoning and could not be answered by purely associative methods.[1]

## Construction

### Topic sources

To get breadth, the authors drew question topics from two sources and then wrote the premises and alternatives themselves.[1]

| Source | What it contributed |
|---|---|
| Randomly selected entries from a corpus of about one million personal stories posted to weblogs in August and September 2008 | Everyday topics, weighted toward social and mental events |
| Subject terms from the Library of Congress Thesaurus for Graphic Materials | People, places and things, with more topics about natural and physical causality |

Source: [1]

For each topic the authors asked themselves what causes it and what it causes, treated the answer as a causal bridging inference (for example, "injuries to the head cause unconsciousness"), and wrote a premise and correct alternative as the two events of that relation. Either event could serve as the premise, depending on whether the question tested forward or backward reasoning.[1] Obscure subject terms, and terms with no obvious role in causal reasoning, were discarded.[1]

### Writing guidelines

The authors used Warren and colleagues' 1979 taxonomy of propositions in causal relations (states, events, actions, cognitions, displays, impulses and goals) and tried to keep both alternatives of a question in the same class.[1] To limit the language-processing burden on systems, every premise and alternative was written in the past tense and kept as short as possible. Proper names of people and places were avoided, as were colloquialisms and slang. Personal pronouns and definite determiners were allowed, so systems still have to resolve simple coreference, such as working out that "his dog" in an alternative belongs to "the man" in the premise.[1]

### Validation and splits

Ten volunteers, native English-speaking adults not affiliated with the project, validated the set. Each answered 200 questions, so every question was answered by two people.[1] Agreement was high (Cohen's kappa = 0.965). The raters answered 26 questions differently from the author's intended answer; those questions were removed and replaced with new ones that two additional raters validated.[1] In the final set, every one of the 1,000 questions had been answered as intended by two raters.[1][3]

The question order was randomized to hide shifts in authoring style, and the position of the correct alternative was randomized so that a random baseline answers exactly 50% correctly.[1] The set was then divided into a 500-question development set for tuning and a 500-question test set, and the authors recommended that researchers report results on both.[1]

### Distribution

The COPA package contains the development, test and combined question sets in XML, gold answers, the output of a first-alternative baseline and of the best 2011 baseline, and a Java tool for testing whether two systems differ significantly.[2] The test uses approximate randomization with stratified shuffling, following Noreen (1989).[2][3] The homepage asks researchers not to study the test questions and to evaluate on the test set only once, after all development work is finished.[2] The materials are released under the BSD 2-Clause License.[2]

## Baselines and early systems (2011-2016)

The 2011 paper tested unsupervised baselines built on the idea that causally related events tend to be mentioned close together in text.[1] For a premise $p$ and an alternative $a$ with $N_p$ and $N_a$ content words, the first baseline averages a word-association score over all pairs of content words:

$$\text{score}(p, a) = \frac{1}{N_p N_a} \sum_{u \in p} \sum_{v \in a} \text{assoc}(u, v)$$

and picks the alternative with the higher score, using either pointwise mutual information (PMI) or the Dice coefficient as $\text{assoc}$.[1] Co-occurrence counts came from the English documents of Project Gutenberg (over 42,000 documents, 16 GB of text) within windows of 5, 25 or 50 words; for backward questions the word order of the association measure was reversed.[1] Two further baselines replaced corpus counts with hit counts from the Bing, Google and Yahoo search engines, at the word-pair level and at the whole-phrase level.[1] The best result was PMI over Project Gutenberg with a 5-word window, at 58.8% on the test set. Web hit counts at the word-pair level were not significantly better than chance on the test set, while two phrase-level variants (the Dice coefficient with Bing or Yahoo hit counts) were.[1] The authors concluded that simple associative methods perform "only moderately above the random baseline."[1]

### SemEval-2012 Task 7

COPA was run as Task 7 of the 6th International Workshop on Semantic Evaluation (SemEval-2012), organized by Andrew Gordon, Zornitsa Kozareva and Melissa Roemmele.[3] Unlike most shared tasks, it supplied no training data, only the development and test questions, and participants could use any text corpus or knowledge source.[3] Only one team completed the task during the two-week evaluation period: Travis Goodwin, Bryan Rink, Kirk Roberts and Sanda Harabagiu of the University of Texas at Dallas Human Language Technology Research Institute.[3] Their two systems were a bigram PMI model built on 8.4 million documents of the LDC Gigaword corpus, and a [support vector machine](https://aiwiki.ai/wiki/support_vector_machine_svm) that combined that PMI with temporal-link statistics, word polarity and cause-effect pairs extracted with dependency patterns.[3]

| System | Year | Test accuracy |
|---|---|---|
| PMI over Project Gutenberg (window 5) | 2011 | 58.8% |
| UTDHLT Bigram PMI (SemEval-2012) | 2012 | 61.8% |
| UTDHLT SVM Combined (SemEval-2012) | 2012 | 63.4% |
| PMI over nearly 1 million weblog stories (window 25) | 2011 | 65.2% |
| PMI over 10.4 million weblog stories (window 25) | 2011 | 65.4% |
| Luo et al., "Commonsense Causal Reasoning between Short Texts" | 2016 | 70.2% |

Sources: [1][2][3]

The task paper noted that the differences between the SemEval systems and the earlier weblog-story systems were not statistically significant, and that with the best systems in the 60-65% range, the field remained "much closer to random performance (50%) than human performance (99%)."[3] The organizers wrote that the results cast some doubt on whether the information needed to answer COPA questions could be readily obtained from large text corpora, and suggested that systems might need automated reasoning that treats each premise as a new world state.[3]

## COPA in SuperGLUE

In 2019 COPA became one of the eight tasks of SuperGLUE, a harder successor to the [GLUE benchmark](https://aiwiki.ai/wiki/glue_benchmark) built by researchers at New York University, Facebook AI Research, the University of Washington and DeepMind.[4] SuperGLUE's version of COPA has 400 training, 100 development and 500 test examples, is scored by accuracy, and lists its text sources as "blogs, photography encyclopedia."[4]

For the [BERT](https://aiwiki.ai/wiki/bert) baseline, each answer choice was concatenated with the context, passed through BERT, and projected to a scalar; the choice with the highest score was taken as the answer.[4] A variant called BERT++ was first fine-tuned on the SWAG dataset because of its similarity to COPA. The paper's text describes this transfer step as an 8-point improvement, although its test-set table shows a smaller gain, from 70.6 to 73.8.[4] Human performance was estimated by having Amazon Mechanical Turk workers re-annotate a sample of the test set, with a majority vote across annotations.[4]

| System (SuperGLUE paper, 2019) | COPA test accuracy |
|---|---|
| Most frequent class | 50.0 |
| CBoW (GloVe bag of words) | 51.6 |
| BERT (bert-large-cased) | 70.6 |
| BERT++ (with SWAG transfer) | 73.8 |
| Best known result, May 2019 (Sap et al.) | 84.4 |
| Human estimate | 100.0 |

Source: [4]

The SuperGLUE authors expected the human-machine gap to be hard to close, noting that "On WSC and COPA, human performance is perfect."[4] The best outside result in their table came from Sap and colleagues' Social IQa paper, which fine-tuned BERT on social commonsense questions before fine-tuning on COPA.[4][2]

## Annotation artifacts and Balanced COPA

In 2019, Pride Kavumba and colleagues at Tohoku University and the RIKEN Center for Advanced Intelligence Project asked whether the gains of BERT and [RoBERTa](https://aiwiki.ai/wiki/roberta) on COPA reflected causal reasoning or the exploitation of superficial cues.[5] They found that individual tokens were unevenly distributed between correct and wrong alternatives. The word "a", for instance, appeared in one alternative but not the other in 21.2% of training instances, and in those instances it was in the correct alternative 57.5% of the time.[5] A RoBERTa model trained only on the two alternatives, with no premise at all, reached a mean accuracy of 59.6%, above the 50% that should be possible without the premise.[5]

The authors split the COPA test set into an "Easy" subset of 190 instances that a premise-free model could solve and a "Hard" subset of the remaining 310.[5] Fine-tuned BERT scored 83.9% on Easy instances but 71.9% on Hard ones, while the gap was smaller for RoBERTa.[5] To remove the cues they built **Balanced COPA**: five fluent English speakers with NLP background wrote a new premise for each of the 500 training instances so that the originally wrong alternative became the right one, giving 1,000 training instances in which every alternative is correct exactly once and wrong exactly once.[5] For example, the original premise "The stain came out of the shirt" (cause: "I bleached the shirt") was mirrored with "The shirt did not have a hole anymore" (cause: "I patched the shirt").[5]

| Finding (Kavumba et al., 2019) | Value |
|---|---|
| RoBERTa-large accuracy with alternatives only | 59.6 (plus or minus 2.3) |
| Fine-tuned BERT, Easy subset | 83.9% |
| Fine-tuned BERT, Hard subset | 71.9% |
| Human accuracy on 100 original COPA instances | 100.0% (Fleiss' kappa 0.973) |
| Human accuracy on 100 Balanced COPA instances | 97.0% (Fleiss' kappa 0.798) |

Source: [5]

The authors concluded that BERT relies on superficial cues when they are present but still reaches comparable accuracy when trained on Balanced COPA, which suggests it learns the task to some degree when forced to, and that RoBERTa does not appear to rely on such cues.[5] The paper appeared at the First Workshop on Commonsense Inference in Natural Language Processing (COIN) in November 2019.[5]

## XCOPA

**XCOPA** (Cross-lingual Choice of Plausible Alternatives), presented at EMNLP 2020 by Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić and Anna Korhonen of the University of Cambridge and the University of Mannheim, translates and re-annotates COPA into 11 languages.[6] The languages were chosen for typological diversity, one from each of 11 language families across five geographical macro-areas: Estonian, Haitian Creole, Indonesian, Italian, Eastern Apurímac Quechua, Kiswahili, Tamil, Thai, Turkish, Vietnamese and Mandarin Chinese.[6] Each language has 100 validation and 500 test instances translated from the English COPA validation and test sets.[6] Translators labeled each item before translating it, and inter-translator agreement was very high (Fleiss' kappa 0.921 on validation and 0.911 on test data).[6]

The XCOPA experiments compared multilingual encoders such as [XLM-RoBERTa](https://aiwiki.ai/wiki/xlm-roberta) and multilingual BERT with a "translate test" approach that machine-translates the test data into English. The translation-based setup with a monolingual English RoBERTa encoder beat the other approaches in every language except Thai.[6] When premises were hidden, performance fell to not much better than chance, which the authors took as evidence that solving the task requires reasoning over the premise.[6] The [lm-evaluation-harness](https://aiwiki.ai/wiki/lm_evaluation_harness) includes XCOPA as a group of per-language tasks.[12]

## Other derived datasets

COPA items have been reused in later datasets beyond Balanced COPA and XCOPA:

| Dataset | Year | What it does |
|---|---|---|
| DIALECT-COPA | 2024 | Translations of COPA into three South Slavic dialects (the Cerkno dialect of Slovenian, the Chakavian dialect of Croatian and the Torlak dialect of Serbian), made by native dialect speakers from the English original and the corresponding standard-language translation |
| ProbCOPA | 2026 | 210 handcrafted probabilistic inferences in English derived from COPA items, each rated for likelihood by 25 to 30 human participants |

Sources: [13][14]

The DIALECT-COPA authors describe their sets as the first resource for evaluating large language models on South Slavic dialects.[13] ProbCOPA, published in Transactions of the Association for Computational Linguistics in 2026, splits each COPA item into two NLI-style items so that each alternative is judged against the premise on its own. Its authors compared human judgments with those of eight reasoning LLMs, including GPT-5, Gemini 3 and Claude Sonnet 4.5, and found that the models consistently failed to produce human-like distributions of judgments.[14]

## Results with large language models

### Few-shot GPT-3

The [GPT-3](https://aiwiki.ai/wiki/gpt-3) paper (2020) evaluated the model on SuperGLUE with [few-shot learning](https://aiwiki.ai/wiki/few-shot_learning), giving it 32 examples in context per task and no gradient updates.[7] On the COPA test set GPT-3 few-shot scored 92.0, against 70.6 for fine-tuned BERT-Large and 94.8 for the fine-tuned state of the art.[7] The authors noted that COPA fell "only a couple points short" of the state of the art and took second place on the leaderboard, behind a fine-tuned 11-billion-parameter [T5](https://aiwiki.ai/wiki/t5) model.[7]

### Fine-tuned models near the ceiling

SuperGLUE test results reported by the [DeBERTa](https://aiwiki.ai/wiki/deberta) authors, taken from the SuperGLUE leaderboard on January 6, 2021, show how quickly the remaining gap shrank:[8]

| System | COPA test accuracy |
|---|---|
| RoBERTa-large | 90.6 |
| NEXHA-Plus | 93.6 |
| T5 11B | 94.8 |
| DeBERTa 1.5B + SiFT | 96.8 |
| T5 11B + Meena | 97.4 |
| DeBERTa ensemble | 98.4 |
| Human | 100.0 |

Source: [8]

In 2022 ST-MoE-32B, a sparse [mixture of experts](https://aiwiki.ai/wiki/mixture_of_experts) model from Google Brain and Google Research, reported 99.2 on the COPA test set after fine-tuning; the paper's comparison table lists 98.2 as the previous fine-tuned best.[9]

### Open models and the authors' 2024 reassessment

In a 2024 paper in Findings of EMNLP, Roemmele (by then at Midjourney) and Gordon evaluated eleven open [large language models](https://aiwiki.ai/wiki/large_language_model) on the COPA test set with a 4-shot prompt (four development-set items as exemplars) and greedy decoding.[10]

| Model | 4-shot COPA test accuracy |
|---|---|
| [Llama 2](https://aiwiki.ai/wiki/llama_2) 70B | 0.976 |
| [Mistral 7B](https://aiwiki.ai/wiki/mistral_7b) | 0.938 |
| Phi-2 | 0.902 |
| Llama 2 13B | 0.850 |
| MPT-30B | 0.848 |
| [Falcon](https://aiwiki.ai/wiki/falcon) 40B | 0.844 |
| Llama 2 7B | 0.826 |
| MPT-7B | 0.656 |
| [BLOOM](https://aiwiki.ai/wiki/bloom) 176B | 0.576 |
| Falcon 7B | 0.538 |
| BLOOM 7B | 0.532 |

Source: [10]

Larger models beat smaller ones within each family, but size did not explain the whole ranking: BLOOM-176B, the largest model tested, ranked third lowest, and Phi-2, the smallest, ranked third highest.[10] The authors wrote that Llama 2-70B's near-perfect accuracy "confirms that COPA is no longer as challenging as it previously was and will soon become outdated."[10] They also cautioned that because the COPA test set is available online, the models may have seen the items during training, a form of [data contamination](https://aiwiki.ai/wiki/data_contamination).[10]

The main subject of that paper was the reverse task: prompting LLMs to write new COPA-style items. The authors found that models that answered the original COPA well were also more successful at authoring their own items.[10]

## Use in evaluation suites

Beyond SuperGLUE, COPA is available in standard evaluation tooling. EleutherAI's lm-evaluation-harness defines a `copa` task, tagged `super-glue-lm-eval-v1`, that reads the SuperGLUE copy of the data and scores it as a multiple-choice task by accuracy.[11] The same harness includes the 11 XCOPA languages.[12] The GPT-3 paper's prompt format for COPA joined the premise and each alternative into one sentence with a connective, as in "My body cast a shadow over the grass because the sun was rising," and, as for other multiple-choice tasks in the paper, compared the model's likelihood of each completion.[7]

## Significance and limitations

The XCOPA authors called it "a seminal work on the quantitative evaluation of commonsense reasoning."[6] Its design choices (a forced choice between two plausible alternatives, a balanced answer position, human validation and fixed development and test splits) were aimed at making progress measurable and comparable across research groups.[1][3]

Its limitations are also documented. The dataset is small, with only 500 test questions and, in SuperGLUE, 400 training examples.[4] Kavumba and colleagues showed that the alternatives carry lexical cues that fine-tuned models can exploit without reading the premise.[5] Scores of fine-tuned and few-shot models now sit close to the human ceiling, and the original authors expect the benchmark to become outdated, while noting possible test-set contamination.[9][10] Successor and companion datasets, including Balanced COPA and XCOPA, address some of these issues by removing token cues and by testing transfer to other languages.[5][6]

## See also

- [Commonsense reasoning](https://aiwiki.ai/wiki/commonsense_reasoning)
- [SuperGLUE](https://aiwiki.ai/wiki/superglue)
- [Winograd Schema Challenge](https://aiwiki.ai/wiki/winograd_schema_challenge)
- [WinoGrande](https://aiwiki.ai/wiki/winogrande)
- [HellaSwag](https://aiwiki.ai/wiki/hellaswag)
- [PIQA](https://aiwiki.ai/wiki/piqa)
- [BoolQ](https://aiwiki.ai/wiki/boolq)
- [Natural language inference](https://aiwiki.ai/wiki/natural_language_inference)

## References

1. Roemmele, Melissa; Bejan, Cosmin Adrian; Gordon, Andrew S. "Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning". AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, Stanford University, March 2011. https://asgordon.github.io/publications/AAAI-SPRING11A.PDF
2. Gordon, Andrew S. "Choice of Plausible Alternatives (COPA): An evaluation of commonsense causal reasoning". COPA homepage, accessed November 2026. https://asgordon.github.io/copa.html
3. Gordon, Andrew; Kozareva, Zornitsa; Roemmele, Melissa. "SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning". *SEM 2012 / Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 394-398, 2012. https://aclanthology.org/S12-1052/
4. Wang, Alex; Pruksachatkun, Yada; Nangia, Nikita; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems". Advances in Neural Information Processing Systems 32 (NeurIPS 2019); arXiv:1905.00537. https://arxiv.org/abs/1905.00537
5. Kavumba, Pride; Inoue, Naoya; Heinzerling, Benjamin; Singh, Keshav; Reisert, Paul; Inui, Kentaro. "When Choosing Plausible Alternatives, Clever Hans can be Clever". Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 33-42, November 2019. https://aclanthology.org/D19-6004/
6. Ponti, Edoardo Maria; Glavaš, Goran; Majewska, Olga; Liu, Qianchu; Vulić, Ivan; Korhonen, Anna. "XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning". Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, November 2020. https://aclanthology.org/2020.emnlp-main.185/
7. Brown, Tom B. et al. "Language Models are Few-Shot Learners". arXiv:2005.14165, 2020. https://arxiv.org/abs/2005.14165
8. He, Pengcheng; Liu, Xiaodong; Gao, Jianfeng; Chen, Weizhu. "DeBERTa: Decoding-enhanced BERT with Disentangled Attention". ICLR 2021; arXiv:2006.03654. https://arxiv.org/abs/2006.03654
9. Zoph, Barret et al. "ST-MoE: Designing Stable and Transferable Sparse Expert Models". arXiv:2202.08906, 2022. https://arxiv.org/abs/2202.08906
10. Roemmele, Melissa; Gordon, Andrew. "From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items". Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5193-5203, November 2024. https://aclanthology.org/2024.findings-emnlp.299/
11. EleutherAI. "lm_eval/tasks/super_glue/copa/default.yaml". lm-evaluation-harness, GitHub, accessed November 2026. https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/super_glue/copa/default.yaml
12. EleutherAI. "XCOPA" task README. lm-evaluation-harness, GitHub, accessed November 2026. https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/xcopa/README.md
13. Ljubešić, Nikola; Galant, Nada; Benčina, Sonja; Čibej, Jaka; Milosavljević, Stefan; Rupnik, Peter; Kuzman, Taja. "DIALECT-COPA: Extending the Standard Translations of the COPA Causal Commonsense Reasoning Dataset to South Slavic Dialects". Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 89-98, 2024. https://aclanthology.org/2024.vardial-1.7/
14. Kamath, Gaurav; Madathil, Sreenath; Schuster, Sebastian; de Marneffe, Marie-Catherine; Reddy, Siva. "Humans and LLMs Diverge on Probabilistic Inferences". Transactions of the Association for Computational Linguistics, 14:2057-2080, 2026. https://aclanthology.org/2026.tacl-1.93/

