Low-Resource Languages

Low-resource languages, in natural language processing (NLP), are languages with little of the raw material that modern language technology depends on: digitized text, parallel translation corpora, annotated datasets, and evaluation benchmarks. The label describes data availability, not the language itself or its number of speakers. Some languages spoken by tens of millions of people, such as Swahili or Bhojpuri, count as low-resource because so little of their use is captured in machine-readable form, while Dutch, with far fewer speakers, sits near the well-resourced end of the scale. Of the more than 7,000 languages spoken worldwide, only a small fraction is represented in NLP research and products at all ^[1].

The gap has practical consequences. Machine translation, speech recognition, search, and large language models (LLMs) all degrade sharply outside the handful of languages that dominate the web, and speakers of everything else get worse output at higher cost. A 2020 study by Joshi et al. sorted the world's languages into six classes by resource availability and found that 2,191 languages, 88.38% of the total, sit in class 0, the "Left-Behinds," with essentially no labeled or unlabeled data; those languages are spoken by roughly 1.2 billion people. At the other end, the seven "Winner" languages (0.28% of all languages, including English, Spanish, German, Japanese, and French) enjoy nearly all of the field's data and tooling ^[1].

Work on low-resource NLP spans several fronts: transfer techniques that reuse knowledge from high-resource languages, data creation and mining efforts, dedicated models such as Meta's NLLB-200, community initiatives such as Masakhane and the Aya project, and benchmarks such as FLORES-200 that make progress measurable. Since 2022 the field has shifted from machine translation as the flagship task toward the broader question of why LLMs serve most of the world's languages so unevenly.

The resource hierarchy

Joshi et al.'s taxonomy, presented at ACL 2020, remains the standard framing of the problem. It scores languages on two axes, the amount of unlabeled text available for pretraining and the amount of labeled data available for supervised tasks, and groups them into six classes ^[1]:

Scroll sideways for more →

The authors noted that class 0 alone accounts for about 15% of the world's speakers, and warned that the lack of technology for these languages could push their speakers toward dominant languages, worsening the imbalance ^[1].

Quantity is only half the problem. A 2022 audit led by Julia Kreutzer and Isaac Caswell manually reviewed 205 language-specific subsets of five widely used web-crawled corpora (CCAligned, ParaCrawl, WikiMatrix, OSCAR, and mC4) and found that at least 15 of them contained no usable text at all, while a significant fraction contained less than 50% acceptable-quality sentences. Mislabeled languages and nonstandard language codes were widespread, and the problems fell disproportionately on the lowest-resourced languages, meaning that even the data that nominally exists for them is often unusable or wrong ^[2].

Why models underperform

Three mechanisms explain most of the performance gap. The first is simple scarcity: neural systems are data-hungry, and for most languages the parallel and monolingual text needed to train them does not exist in digital form ^[1]. The second is data quality, as the Kreutzer audit showed: scraping the web for a language with a small online footprint mostly yields noise, boilerplate, or text in some other language wearing the wrong language code ^[2].

The third is tokenization. Subword tokenizers such as byte-pair encoding build their vocabularies from training corpora dominated by English, so text in underrepresented languages fragments into many more tokens. Petrov et al. showed at NeurIPS 2023 that the same passage translated into different languages can differ in tokenized length by up to 15 times, and that even character-level and byte-level encodings show disparities of over 4x for some language pairs. Since commercial APIs bill by the token, and context windows and latency are also measured in tokens, speakers of these languages pay more for less ^[3]. A companion analysis by Ahia et al. across 22 typologically diverse languages found that users of many languages supported by commercial APIs are "overcharged while obtaining poorer results," and that the overcharged populations tend to come from regions where the APIs are less affordable to begin with ^[4].

Transfer techniques

Multilingual pretraining

The dominant strategy since 2018 has been to pretrain one model on many languages at once, letting the low-resource ones borrow statistical strength from their better-documented neighbors. Multilingual BERT (mBERT), released by Google in November 2018, was trained on Wikipedia text in 104 languages with a shared 110,000-piece WordPiece vocabulary, using exponential smoothing to oversample small languages and undersample English ^[5]. XLM-R (XLM-RoBERTa), published by Facebook AI researchers in 2019, scaled the recipe to more than two terabytes of filtered Common Crawl text in 100 languages and improved average XNLI accuracy by 14.6% over mBERT, with outsized gains for low-resource languages: 15.7% for Swahili and 11.4% for Urdu over earlier XLM models ^[6]. Later encoder-decoder models such as mT5 followed the same pattern.

Capacity is the catch. A single model splits its parameters across every language it covers, so per-language performance drops as coverage grows; Google's own documentation reports mBERT scoring about 3% below a Chinese-only BERT on Chinese tasks ^[5]. The trade-off between breadth and depth remains a central design question for multilingual models.

Cross-lingual transfer

Multilingual pretraining enables cross-lingual transfer learning: fine-tuning a model on labeled data in a high-resource language, usually English, and applying it zero-shot to languages that have no labeled data at all. This is what makes tasks like named entity recognition or question answering feasible for languages with no annotated training sets. Evaluations on the Belebele reading-comprehension benchmark found that English-centric LLMs do transfer significantly across languages, but that much smaller masked language models pretrained on balanced multilingual data still understand far more languages, and that larger, deliberately constructed vocabularies correlate with better low-resource performance ^[7].

Back-translation and data augmentation

For translation specifically, the standard data augmentation trick is back-translation, introduced for neural machine translation by Sennrich, Haddow, and Birch at ACL 2016. A reverse-direction system translates monolingual target-language text into the source language, and the resulting synthetic pairs are added to the training data. The technique gained 2.8 to 3.7 BLEU on WMT 15 English-German and 2.1 to 3.4 BLEU on the low-resourced IWSLT 14 Turkish-English task, and it remains a staple wherever monolingual text is easier to find than parallel text ^[8].

Data mining and creation

The complementary approach is to manufacture the missing data. Meta's NLLB project mined parallel sentences from the web at scale using LASER3 multilingual sentence embeddings and commissioned professional translations (NLLB-Seed) for languages with almost nothing ^[9][10]. Google's MADLAD-400, released in 2023, took the quality lesson seriously: a manually audited monolingual dataset covering 419 languages and 3 trillion tokens from Common Crawl, used to train a 10.7 billion parameter translation model covering more than 450 languages ^[11]. Community annotation is the third route, discussed below.

Flagship projects

Scroll sideways for more →

No Language Left Behind

Meta announced NLLB-200 in July 2022 as a single translation model covering 200 languages. The main model is a Sparsely Gated Mixture-of-Experts with 54.5 billion parameters, released openly alongside a dense 3.3 billion parameter variant and distilled 1.3 billion and 600 million parameter versions ^[9][10]. The project defined high-resource languages as those with at least 1 million aligned sentence pairs; roughly 150 of the 200 languages fell below that line ^[10]. Against the prior state of the art, NLLB-200 improved BLEU by 44% on average, with gains above 70% for some African and Indian languages ^[9]. The accompanying FLORES-200 benchmark, professionally translated across all 200 languages, made evaluation possible over more than 40,000 translation directions ^[9][10]. The work was published in Nature in June 2024 ^[10], and the benchmark now continues as FLORES+ under the Open Language Data Initiative ^[18]. Through a partnership with the Wikimedia Foundation, NLLB-200 also backs Wikipedia's Content Translation tool for more than 20 low-resource languages ^[9].

Masakhane

Masakhane (roughly "we build together" in isiZulu) is a grassroots research collective working on NLP "for Africans, by Africans." Africa's roughly 2,000 languages are among the most poorly served by language technology, and Masakhane's answer has been participatory research: its community of more than 1,000 members across 30 African countries includes translators, linguists, and self-taught contributors without formal NLP training ^[13]. Its foundational paper, published at Findings of EMNLP 2020 with 48 authors, argued that "low-resourcedness" reflects systemic issues beyond data scarcity and demonstrated the participatory model by producing translation datasets for over 30 African languages, with human-evaluated benchmarks for about a third of them ^[14]. Follow-on projects include MasakhaNER for named entity recognition and speech and text efforts across East and West Africa ^[13].

Aya

Aya is an open multilingual effort led by Cohere For AI. The Aya model, released in February 2024, is an instruction-tuned LLM that follows instructions in 101 languages, over half of them lower-resourced; it outperformed the earlier open multilingual models mT0 and BLOOMZ on most tasks while covering roughly twice as many languages, with weights published on Hugging Face under an Apache 2.0 license ^[15]. The companion Aya Dataset, built by contributors across 119 countries through a purpose-built annotation platform, is a human-curated instruction-following collection spanning 65 languages, extended to 513 million instances across 114 languages through templating and translation ^[16]. Cohere followed up with Aya Expanse (8B and 32B), described in a December 2024 paper, which combined multilingual preference training, data arbitrage, and model merging; the 32B model beat Llama 3.1 70B on the m-ArenaHard multilingual evaluation with a 54.0% win rate despite having fewer than half the parameters, across 23 languages ^[19].

Speech: Omnilingual ASR

Text is not the only modality with a long tail. In November 2025 Meta released Omnilingual ASR, a family of open speech recognition models (Apache 2.0) covering more than 1,600 languages, including 500 low-resource languages that had never previously been transcribed by an AI system. The release included the Omnilingual ASR Corpus, spontaneous speech data for 350 underserved languages under CC-BY, and a mechanism for extending the system to new languages from just a handful of paired audio-text examples. Meta reported character error rates below 10 for 78% of supported languages ^[17].

Benchmarks and their gaps

Evaluation lags even further behind modeling. FLORES-200 and Belebele, a reading-comprehension benchmark built from FLORES-200 passages with four-way multiple-choice questions in 122 language variants, are among the few tests that cover a genuinely wide slice of the world's languages ^[7][10].

Coverage aside, translated benchmarks import cultural assumptions. Global MMLU, a 42-language re-annotation of the widely used MMLU test released by Cohere For AI and collaborators in December 2024, found that 28% of MMLU questions require culturally sensitive knowledge that skews Western, and that among questions needing geographic knowledge, 84.9% concern North America or Europe. Model rankings changed materially depending on whether the culturally sensitive subset was included, which undercuts naive "translate and evaluate" pipelines ^[20].

Benchmarks built by and for the affected communities paint a bleaker picture of current LLMs. IrokoBench (NAACL 2025) evaluates 17 typologically diverse African languages on natural language inference, mathematical reasoning, and MMLU-style knowledge questions. It found a wide gap between performance on high-resource languages such as English and French and on African languages, and a second gap between open and proprietary models: the best open model tested, Gemma 2 27B, reached only about 63% of GPT-4o's performance. Translating test items into English before answering narrowed the gap for large English-centric models, a sign that the underlying knowledge exists but is locked behind the language barrier ^[21].

Outlook

The through-line of the last few years is that low-resource NLP has moved from a translation problem to an ecosystem problem: pretraining data, tokenizers, instruction data, evaluation, and pricing all encode the same imbalance, and fixing any one of them alone helps only so much ^[2][3][20]. The most effective recent efforts have paired large industrial releases (NLLB-200, MADLAD-400, Omnilingual ASR) with community-driven data work (Masakhane, Aya, the Open Language Data Initiative), the former supplying scale and the latter supplying the language expertise and quality control that web scraping cannot ^{[11][13][16][18]}. Whether LLM progress narrows or widens the gap is still an open question; benchmarks like IrokoBench and Global MMLU at least make the answer measurable ^[20][21].

See also

• No Language Left Behind
• FLORES-200
• Machine translation
• Neural machine translation
• Aya Expanse
• XLM-RoBERTa

References

1. ^Joshi, P., Santy, S., Budhiraja, A., Bali, K., Choudhury, M. "The State and Fate of Linguistic Diversity and Inclusion in the NLP World." ACL 2020. aclanthology.org/2020.acl-main.560
2. ^Kreutzer, J., Caswell, I., et al. "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets." TACL, 2022. arxiv.org/...2103.12028
3. ^Petrov, A., La Malfa, E., Torr, P., Bibi, A. "Language Model Tokenizers Introduce Unfairness Between Languages." NeurIPS 2023. arxiv.org/...2305.15425
4. ^Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D., Smith, N., Tsvetkov, Y. "Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models." 2023. arxiv.org/...2305.13707
5. ^Google Research. "BERT Multilingual Models" (multilingual.md), November 2018. github.com/...multilingual.md
6. ^Conneau, A., Khandelwal, K., Goyal, N., et al. "Unsupervised Cross-lingual Representation Learning at Scale." ACL 2020. arxiv.org/...1911.02116
7. ^Bandarkar, L., Liang, D., Muller, B., et al. "The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants." ACL 2024. arxiv.org/...2308.16884
8. ^Sennrich, R., Haddow, B., Birch, A. "Improving Neural Machine Translation Models with Monolingual Data." ACL 2016. arxiv.org/...1511.06709
9. ^Meta AI. "200 languages within a single AI model: A breakthrough in high-quality machine translation." July 2022. ai.meta.com/...00-high-quality-machine-translation
10. ^NLLB Team et al. "Scaling neural machine translation to 200 languages." Nature 630, 841-846, June 5, 2024. nature.com/...s41586-024-07335-x (preprint: arxiv.org/...2207.04672)
11. ^Kudugunta, S., Caswell, I., Zhang, B., et al. "MADLAD-400: A Multilingual And Document-Level Large Audited Dataset." 2023. arxiv.org/...2309.04662
12. ^Guzman, F., Chen, P.-J., Ott, M., et al. "The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English." EMNLP 2019. arxiv.org/...1902.01382
13. ^Masakhane. Official website. masakhane.io
14. ^Nekoto, W., Marivate, V., et al. "Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages." Findings of EMNLP 2020. arxiv.org/...2010.02353
15. ^Ustun, A., Aryabumi, V., Yong, Z.-X., et al. "Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model." February 2024. arxiv.org/...2402.07827
16. ^Singh, S., et al. "Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning." February 2024. arxiv.org/...2402.06619
17. ^Meta AI. "Omnilingual ASR: Advancing automatic speech recognition for 1,600+ languages." November 10, 2025. ai.meta.com/...ancing-automatic-speech-recognition
18. ^Open Language Data Initiative. Official website. oldi.org
19. ^Dang, J., et al. "Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier." December 2024. arxiv.org/...2412.04261
20. ^Singh, S., Romanou, A., Fourrier, C., Adelani, D. I., et al. "Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation." December 2024. arxiv.org/...2412.03304
21. ^Adelani, D. I., et al. "IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models." NAACL 2025. arxiv.org/...2406.03368