Neural Machine Translation

Neural machine translation (NMT) is an approach to machine translation in which a single artificial neural network, trained end to end on bilingual text, maps a sentence in one language to a sentence in another. It replaced statistical machine translation (SMT), which assembled translations from separately trained components such as phrase tables, language models, and reordering models. An NMT system instead learns one set of parameters that reads the source sentence and generates the target sentence word by word (or subword by subword), usually with beam search at decoding time.

The approach went from research prototype to production remarkably fast. The first modern neural translation models appeared in 2013 and 2014, Google moved its translation service onto neural networks in 2016, and the Transformer architecture introduced in 2017 became the standard for essentially all subsequent systems ^[6][7][10]. NMT is also the direct ancestor of today's large language models: the Transformer was designed for translation first, and concepts such as attention, subword tokenization, and encoder-decoder structure all matured inside machine translation research before spreading across natural language processing.

Since about 2023 the field has been reshaped again by general-purpose LLMs, which now translate high-resource language pairs at a level competitive with dedicated translation systems, while evaluation campaigns keep finding that translation is not a solved problem, particularly for low-resource languages and document-level coherence ^[21][22][23].

From statistical to neural translation

Through the 2000s and early 2010s, the dominant paradigm was phrase-based SMT: systems memorized phrase pairs extracted from parallel corpora and scored candidate translations with a pipeline of separate models. Neural networks entered this pipeline gradually. In 2013, Nal Kalchbrenner and Phil Blunsom published Recurrent Continuous Translation Models, an early attempt to generate translations from continuous representations with a recurrent neural network rather than from discrete phrase tables ^[1]. In June 2014, Kyunghyun Cho, Yoshua Bengio, and colleagues introduced the RNN encoder-decoder: one recurrent network compresses the source sentence into a fixed-length vector, and a second network generates the target sentence from that vector. The paper, which also introduced the gated recurrent unit (GRU), still used the neural model as a feature inside an SMT system rather than as the translator itself ^[2].

The clean break came in September 2014 with Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, and Quoc V. Le at Google. Their model used one multilayer LSTM to encode the source sentence into a vector and another deep LSTM to decode the translation, with no SMT machinery at all. On the WMT'14 English-to-French task it reached 34.8 BLEU against 33.3 for a phrase-based baseline, and 36.5 when used to rerank the baseline's hypotheses. The authors also found that simply reversing the word order of source sentences improved results markedly, because it shortened the dependencies the optimizer had to learn ^[3].

Attention

The fixed-length vector was the weak point of early sequence-to-sequence models: every source sentence, however long, had to be squeezed through the same bottleneck. In a paper posted in September 2014 and presented at ICLR 2015, Dzmitry Bahdanau, Cho, and Bengio identified this bottleneck explicitly and proposed letting the decoder "(soft-)search" the source sentence, computing a fresh weighted combination of encoder states for every target word it generates ^[4]. This mechanism, now known as attention, let the model learn alignments between source and target words as a byproduct of translation, made performance on long sentences far more robust, and brought a pure neural system up to the level of phrase-based SMT on English-to-French ^[4].

Attention turned out to be the most consequential idea in the NMT line of work. Within three years it had been generalized into an architecture built from nothing else.

Subword vocabularies

Early NMT systems worked with fixed word vocabularies of perhaps 30,000 to 80,000 entries, and anything outside the vocabulary became an unknown token. Rico Sennrich, Barry Haddow, and Alexandra Birch solved this in a 2016 ACL paper by adapting byte pair encoding (BPE), a 1990s compression algorithm, into a tokenization scheme: frequent words stay whole, rare words split into smaller subword units, and the system can compose translations of words it has never seen, including names and morphological variants. Subword models improved over back-off dictionary baselines by 1.1 BLEU on English-to-German and 1.3 BLEU on English-to-Russian ^[5]. BPE and close relatives such as Google's wordpieces became universal in NMT, and later in large language models, where BPE-style tokenizers remain standard.

Into production

Google announced on September 27, 2016 that its production translation service was switching to neural translation, starting with Chinese to English, a pair that at the time accounted for roughly 18 million translations per day ^[7]. The underlying system, Google Neural Machine Translation (GNMT), was described in a paper by Yonghui Wu, Mike Schuster, and colleagues: a deep LSTM network with 8 encoder layers and 8 decoder layers, residual connections, attention from decoder to encoder, and a wordpiece subword vocabulary. In human side-by-side evaluations on sampled sentences, GNMT reduced translation errors by an average of 60 percent relative to Google's phrase-based production system ^[6]. Google's announcement was candid about remaining failure modes, including dropped words and mistranslated proper names ^[7], early examples of the fluent-but-wrong behavior later studied under the heading of hallucination.

Recurrent networks were not the only option. In May 2017 researchers at Facebook AI Research published a fully convolutional sequence-to-sequence model: Jonas Gehring, Michael Auli, and colleagues showed that a convolutional neural network with gated linear units and attention in every decoder layer could beat GNMT's accuracy on WMT'14 English-German and English-French while running about an order of magnitude faster, since convolutions parallelize over the whole sequence during training ^[8]. The commercial translator DeepL, launched in late August 2017 by the team behind the dictionary site Linguee, also built its first system on convolutional networks with attention rather than on LSTMs ^[9].

The Transformer

The convolutional detour was brief. In June 2017, Ashish Vaswani and seven co-authors at Google released Attention Is All You Need, which removed recurrence and convolution entirely and built the encoder and decoder from stacked self-attention and feed-forward layers. On WMT 2014 the Transformer reached 28.4 BLEU on English-to-German, more than 2 BLEU above the best previous results including ensembles, and a then state-of-the-art single-model score of 41.8 BLEU on English-to-French, after training for 3.5 days on eight GPUs, a fraction of the compute used by earlier systems ^[10].

The Transformer became the default architecture for machine translation within about a year of publication and has stayed there. Its larger legacy lies outside translation: BERT, the GPT series, and effectively every modern large language model is a Transformer variant, which makes NMT the field where the architecture of the current AI wave was invented.

Multilingual models and zero-shot translation

Classic NMT trained one model per language pair. In November 2016, Melvin Johnson, Schuster, and colleagues at Google showed that a single model could serve many pairs at once: the only change needed was an artificial token at the start of the input specifying the desired target language, with the encoder, decoder, and attention shared across all languages. The shared model enabled transfer between related pairs and, more strikingly, zero-shot translation: usable translation between pairs the model had never seen in training, such as between two languages that had each only been paired with English ^[11].

Multilingual NMT scaled up dramatically from there. Meta AI's No Language Left Behind project, published in July 2022, trained NLLB-200, a sparsely gated mixture-of-experts model covering 200 languages, using data mining techniques aimed specifically at low-resource pairs. The team reported a 44 percent average BLEU improvement over the prior state of the art, evaluated translations across more than 40,000 direction pairs, released the FLORES-200 human-translated benchmark covering all of its languages, and open-sourced the models and data pipeline ^[12]. Meta AI followed in August 2023 with SeamlessM4T, a single multimodal model handling speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation plus speech recognition for up to 100 languages ^[13].

Open model ecosystems grew in parallel. The Helsinki-based OPUS-MT project has published over 1,000 pretrained translation models built on the Marian toolkit, free to download and light enough to run on desktop hardware ^[14]. On the commercial side, Google used the PaLM 2 large language model to add 110 languages to Google Translate in June 2024, its largest single expansion, including long-requested languages such as Cantonese ^[15].

Data and the low-resource problem

NMT is data hungry, and its quality tracks the amount of parallel text available for a language pair. The most influential technique for stretching limited data is back-translation, an idea with roots in statistical MT that Sennrich, Haddow, and Birch brought to NMT in 2016: translate target-language monolingual text backwards into the source language with an existing model, then treat the synthetic pairs as extra training data. Back-translation added 2.8 to 3.7 BLEU on English-German and 2.1 to 3.4 BLEU on Turkish-English benchmarks and remains standard practice ^[16].

Even so, the gap between high-resource and low-resource languages is the field's most persistent problem. Most of the world's roughly 7,000 languages have little or no parallel text on the web, which is why NLLB invested as much in data mining and in its toxicity and quality evaluations as in the model itself ^[12]. Evaluations consistently show quality falling off sharply outside the best-resourced few dozen languages, for LLM translators as much as for dedicated NMT systems ^[21].

Evaluation

Machine translation runs on automatic metrics, and their history mirrors the field's. The table lists the three most common today.

Scroll sideways for more →

BLEU dates from the phrase-based era and compares surface word overlap; chrF works at the character level, which helps with morphologically rich languages ^[18]; COMET and similar learned metrics reached new state-of-the-art correlation with human quality judgments and are now standard in WMT evaluations ^[19].

Human evaluation has its own pitfalls. In 2018, Samuel Läubli, Sennrich, and Martin Volk revisited claims that NMT had achieved "human parity" and showed the result depended on rating isolated sentences: when raters compared whole documents, they preferred human translations significantly more often, because errors of coherence, consistency, and context only become visible across sentence boundaries ^[20]. That finding pushed the field toward document-level evaluation, and by 2025 the main WMT shared task evaluated systems on full multi-paragraph documents rather than sentences ^[23].

NMT in the LLM era

Large language models translate without being built for it. A February 2023 Microsoft evaluation of GPT models, including ChatGPT, across 18 translation directions found them "very competitive" with state-of-the-art dedicated systems on high-resource languages while clearly weaker on low-resource ones ^[21]. The WMT24 General Machine Translation shared task, whose findings paper was titled "The LLM Era Is Here but MT Is Not Solved Yet," compared participant systems against 8 LLMs and 4 online translation providers across 11 language pairs, using a new Error Span Annotation protocol with professional annotators ^[22].

The WMT25 edition, held in late 2025, evaluated 60 systems over 30 language pairs, including 24 baselines drawn from LLMs and commercial translation providers, and moved human evaluation to whole documents. Its findings paper, subtitled "Time to Stop Evaluating on Easy Test Sets," argued that standard news-style test data had become too easy for frontier systems and introduced difficulty-sampled test sets to keep the benchmark meaningful ^[23]. The practical picture in 2026 is a hybrid one: general LLMs dominate quality rankings for many high-resource pairs, while dedicated NMT models remain relevant where cost, speed, on-device deployment, or coverage of the long tail of languages matters, and open translation models such as NLLB-200 and the OPUS-MT catalogue serve languages that commercial LLM providers cover poorly ^[12][14][23].

Milestones

Scroll sideways for more →

See also

• Machine translation
• Attention Is All You Need
• Bahdanau attention
• Byte pair encoding
• No Language Left Behind
• Low-resource languages

References

1. ^Kalchbrenner, N. and Blunsom, P. "Recurrent Continuous Translation Models." EMNLP 2013. aclanthology.org/D13-1176
2. ^Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y. "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation." EMNLP 2014. arxiv.org/...1406.1078
3. ^Sutskever, I., Vinyals, O., Le, Q.V. "Sequence to Sequence Learning with Neural Networks." NeurIPS 2014 (arXiv September 10, 2014). arxiv.org/...1409.3215
4. ^Bahdanau, D., Cho, K., Bengio, Y. "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR 2015 (arXiv September 1, 2014). arxiv.org/...1409.0473
5. ^Sennrich, R., Haddow, B., Birch, A. "Neural Machine Translation of Rare Words with Subword Units." ACL 2016. arxiv.org/...1508.07909
6. ^Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., et al. "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." arXiv, September 26, 2016. arxiv.org/...1609.08144
7. ^Google Research. "A Neural Network for Machine Translation, at Production Scale." Google Research Blog, September 27, 2016. research.google/...translation-at-production-scale
8. ^Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N. "Convolutional Sequence to Sequence Learning." arXiv, May 8, 2017. arxiv.org/...1705.03122
9. ^Coldewey, D. "DeepL schools other online translators with clever machine learning." TechCrunch, August 29, 2017. techcrunch.com/...ors-with-clever-machine-learning
10. ^Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I. "Attention Is All You Need." NeurIPS 2017 (arXiv June 12, 2017). arxiv.org/...1706.03762
11. ^Johnson, M., Schuster, M., Le, Q.V., Krikun, M., Wu, Y., et al. "Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation." arXiv, November 14, 2016. arxiv.org/...1611.04558
12. ^NLLB Team, Costa-jussà, M.R., Cross, J., et al. "No Language Left Behind: Scaling Human-Centered Machine Translation." arXiv, July 11, 2022. arxiv.org/...2207.04672
13. ^Seamless Communication team. "SeamlessM4T: Massively Multilingual & Multimodal Machine Translation." arXiv, August 22, 2023. arxiv.org/...2308.11596
14. ^Tiedemann, J. and Thottingal, S. "OPUS-MT: Building open translation services for the World." EAMT 2020. aclanthology.org/2020.eamt-1.61
15. ^Google. "110 new languages are coming to Google Translate." The Keyword (Google Blog), June 27, 2024. blog.google/...google-translate-new-languages-2024
16. ^Sennrich, R., Haddow, B., Birch, A. "Improving Neural Machine Translation Models with Monolingual Data." ACL 2016. arxiv.org/...1511.06709
17. ^Papineni, K., Roukos, S., Ward, T., Zhu, W.-J. "Bleu: a Method for Automatic Evaluation of Machine Translation." ACL 2002. aclanthology.org/P02-1040
18. ^Popović, M. "chrF: character n-gram F-score for automatic MT evaluation." WMT 2015. aclanthology.org/W15-3049
19. ^Rei, R., Stewart, C., Farinha, A.C., Lavie, A. "COMET: A Neural Framework for MT Evaluation." EMNLP 2020. aclanthology.org/2020.emnlp-main.213
20. ^Läubli, S., Sennrich, R., Volk, M. "Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation." EMNLP 2018. arxiv.org/...1808.07048
21. ^Hendy, A., Abdelrehim, M., Sharaf, A., et al. "How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation." arXiv, February 18, 2023. arxiv.org/...2302.09210
22. ^"Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet." WMT 2024. aclanthology.org/2024.wmt-1.1
23. ^"Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets." WMT 2025. aclanthology.org/2025.wmt-1.22