Natural language generation

Natural language generation (NLG) is the subfield of natural language processing and artificial intelligence concerned with building systems that produce understandable text in English or other human languages, typically from some underlying non-linguistic representation of information such as a database, a set of sensor readings, or a formal meaning representation ^[1][2]. For most of its history the field was organized around a modular pipeline that split generation into distinct decisions: what to say, how to organize it, and how to phrase it. That framework, codified by Ehud Reiter and Robert Dale in the 1990s, shaped both academic research and a commercial data-to-text industry that automated weather forecasts, earnings reports, and sports recaps ^[1][3][23].

Since around 2019 the picture has changed sharply. Large language models generate fluent open-ended text from a prompt, with no explicit planning or realization stages, and the term "NLG" is now often used loosely for any text that a generative model produces ^[15][16]. The classic concerns of the field did not disappear, though. Faithfulness to input data, controllability, and the weakness of automatic evaluation metrics remain open problems, and they have become more visible, not less, in the era of large models ^[8][21][22].

Definition and scope

The most cited definition comes from Reiter and Dale, who characterized NLG as the construction of computer systems that produce understandable natural language text from non-linguistic representations of information ^[1][2]. Researchers commonly split the wider territory into two families ^[2]:

• Data-to-text generation, where the input is structured or numerical: database records, time series from sensors, box scores, RDF triples from a knowledge graph. Weather forecast generators and automated earnings stories fall in this family.
• Text-to-text generation, where the input is itself text: machine translation, text summarization, paraphrasing, and text simplification. These tasks involve generation but are usually studied as fields of their own.

Gatt and Krahmer's widely used 2018 survey takes the position that what distinguishes NLG proper is the non-linguistic (or not exclusively linguistic) nature of the input ^[2]. NLG is also the natural counterpart of natural language understanding: understanding maps language to meaning, generation maps meaning to language.

The classic pipeline

Reiter observed in 1994 that practical generation systems had converged on a modular design, and Reiter and Dale's 1997 paper and 2000 book Building Natural Language Generation Systems turned that observation into the field's reference architecture ^[1][2]. They decomposed generation into six subtasks ^[1][2]:

Scroll sideways for more →

In the standard three-stage arrangement, a text planner (or document planner) handles content determination and structuring, a sentence planner (or microplanner) handles aggregation, lexicalisation, and referring expressions, and a linguistic realiser applies syntactic and morphological rules to produce the final sentences ^[2]. The first stage is often called strategic generation, deciding what to say; the later stages are tactical generation, deciding how to say it ^[2].

Each stage attracted its own research literature. Referring expression generation, for example, produced a line of algorithms from Dale's Full Brevity and Greedy Heuristic procedures to the Dale and Reiter Incremental Algorithm of 1995, all trying to pick just enough properties to identify an object without rambling ^[2].

Templates, grammars, and realizers

For the realization stage, three broad approaches coexisted for decades ^[2]:

• Templates. A pattern such as "$player scored for $team in the $minute minute" is filled with values. Templates give full control over output quality and cannot produce ungrammatical text, but they are labor intensive to write and scale poorly when texts need variation ^[2].
• Hand-coded grammar-based realizers. General-purpose systems such as FUF/SURGE, MUMBLE, KPML, and RealPro encode a grammar of the language and derive sentences from detailed input specifications ^[2]. Their input requirements proved so demanding that simpler engines emerged; SimpleNLG, a Java library by Albert Gatt and Ehud Reiter presented in 2009, deliberately limits itself to syntax and morphology APIs and leaves choices to the developer ^[5].
• Statistical realization. Irene Langkilde and Kevin Knight's NITROGEN and HALOGEN systems, in work beginning in the late 1990s, used a small hand-crafted grammar to generate a forest of candidate realizations, then picked among them with corpus-derived n-gram statistics; later work such as the OpenCCG surface realizer achieved wide coverage with grammars derived from the Penn Treebank ^[2].

Data-to-text in practice

Data-to-text was the commercial face of classic NLG, and weather was its proving ground. FoG (Forecast Generator), described by Eli Goldberg, Norbert Driedger, and Richard Kittredge in 1994, generated routine weather forecasts from numerical weather prediction data within Environment Canada's forecast production environment, producing output in both English and French ^[3]. A decade later the SumTime project at the University of Aberdeen generated marine forecasts for offshore oil operations, and its detailed study of how forecasters choose words (which verbs describe a wind shift, what "by evening" means) fed directly into the generator's design ^[4][2].

Journalism followed. The Los Angeles Times' Quakebot filled a prewritten template with data from an automatic earthquake feed and got a story about the March 17, 2014 earthquake near Westwood onto the paper's website within about three minutes of the shaking ^[2][28][29]. The same year the Associated Press began automating quarterly corporate earnings stories with Wordsmith, the platform built by Automated Insights, raising its earnings coverage more than tenfold ^[23][28].

Scroll sideways for more →

The vendors' fates trace the industry's arc. Narrative Science grew out of StatsMonkey, a sports-recap prototype from Northwestern University's Intelligent Information Lab, incorporated in January 2010, and shifted from sports stories to its Quill business intelligence product; Salesforce announced its acquisition on November 15, 2021, closed it on December 17, 2021, and folded the technology into Tableau ^[24]. Automated Insights was acquired by Vista Equity Partners in February 2015, and its technology is now marketed through the sports data company Stats Perform ^[23][27]. Arria NLG remains an independent data-to-text vendor and now positions its platform as a hybrid: deterministic data-to-text for facts that must be exactly right, with LLM components layered on for summarization and conversational access ^[25].

Neural NLG

Sequence-to-sequence learning with recurrent neural networks, introduced for neural machine translation by Sutskever, Vinyals, and Le in 2014, gave NLG researchers a way to learn the whole pipeline end to end: an encoder maps the input (a meaning representation, a table, a set of triples) into a vector, and an LSTM decoder emits the text ^[10]. The sequence-to-sequence framing collapsed the classic module boundaries, since a single trained model implicitly performed content selection, aggregation, and realization at once.

Two shared tasks defined the period. The E2E NLG Challenge, built on a crowdsourced restaurant-domain dataset roughly ten times bigger than earlier NLG corpora, drew 62 submitted systems from 17 institutions in 2017 ^[11][12]. Sequence-to-sequence models dominated and scored highest on word-overlap metrics and human naturalness ratings, but the organizers found that they often failed to express the input meaning correctly without an added semantic control mechanism, and that hand-engineered rule and template systems could still beat them on overall quality ^[12]. The parallel WebNLG challenge asked systems to verbalize sets of RDF triples, exercising microplanning decisions such as aggregation, referring expressions, and sentence segmentation ^[13].

Wiseman, Shieber, and Rush pushed the harder case of long documents: generating multi-paragraph basketball game summaries from box score data. Their 2017 study found neural models produced fluent text but failed to convincingly approximate human-written documents, and that simple templated baselines still outperformed neural systems on some measures ^[14].

Because a neural decoder emits a probability distribution over the vocabulary, output quality also came to depend on decoding strategy: greedy decoding, beam search, or stochastic schemes such as temperature and top-p sampling. Controlling what a free-running language model writes became its own research problem, called controllable generation. Salesforce's CTRL (2019) trained a 1.63 billion parameter conditional transformer whose control codes, derived from the structure of the training data, steer style, content, and task behavior ^[18]. The Plug and Play Language Model (PPLM) approach instead left the pretrained model untouched and used gradients from small attribute classifiers to push its hidden activations toward a topic or sentiment during decoding ^[19].

Hallucination and faithfulness

Neural generation traded the rigidity of templates for a new failure mode: fluent text that is not supported by the input. In data-to-text and summarization research this is called hallucination, and surveys distinguish intrinsic hallucinations, which contradict the source material, from extrinsic ones, which add material the source cannot verify ^[21]. Maynez and colleagues found substantial hallucinated content in the output of every neural abstractive summarization model they evaluated in 2020 ^[20].

The problem persists with large models. A 2024 ACL study by Kasner and Dušek tested open LLMs (Llama 2, Mistral, Zephyr) on data-to-text tasks over freshly collected data, and found that more than 80 percent of outputs contained at least one semantic error, even though the texts were fluent ^[22]. For the commercial data-to-text sector this is the central argument for keeping deterministic generation in the loop: a pipeline that computes its statements from the data cannot, by construction, state a number that is not there ^[25]. Grounding techniques and retrieval-augmented generation attack the same problem from the neural side.

Evaluation

NLG evaluation borrowed its automatic metrics from neighboring fields: BLEU, an n-gram precision metric designed in 2002 for machine translation ^[6], and ROUGE, an overlap metric introduced in 2004 for summarization ^[7], along with METEOR and others. Their fit for NLG has always been questioned, because a generation task usually admits many acceptable outputs and word overlap with a handful of references measures none of the things practitioners care most about, such as factual correctness.

Novikova and colleagues made the case empirically in 2017: across data-driven end-to-end NLG systems, popular metrics correlated only weakly with human judgments of output quality, and how well a metric performed depended on the dataset and system under test. They concluded that metrics can help at the system level, for instance to flag poorly performing configurations during development, but are unreliable for judging individual outputs ^[8]. Embedding-based metrics such as BERTScore, which matches tokens by contextual similarity instead of exact strings, correlate better with human judgments but still measure similarity to references rather than correctness ^[9]. Human evaluation, whether intrinsic (ratings of fluency, adequacy, or faithfulness) or extrinsic (whether the text helps its reader accomplish something), remains the standard that automatic metrics are validated against ^[2][8].

The LLM era

OpenAI's GPT-2 (2019), a 1.5 billion parameter transformer trained on web text, showed that a plain language model with no task-specific architecture could generate coherent multi-paragraph text ^[15]. GPT-3 scaled the recipe to 175 billion parameters in 2020 and could perform tasks, including news article generation that human raters struggled to tell from human writing, from a few examples in the prompt ^[16]. ChatGPT, released on November 30, 2022 and trained with reinforcement learning from human feedback, put open-ended generation in front of a mass audience ^[17].

For NLG as a discipline, the consequence was that the pipeline collapsed into a prompt. Content determination, structuring, aggregation, lexicalisation, and realization all happen implicitly inside one model, steered by instruction tuning and prompt engineering rather than by separate modules. Much of what was published as NLG research now appears as general LLM work on text generation, controllability, and evaluation, and dedicated data-to-text systems compete with, or wrap, general-purpose models ^[22][25][26].

The field's accumulated lessons transferred more or less intact. Ehud Reiter's 2024 Springer book Natural Language Generation, written a quarter century after the 2000 Reiter and Dale text, treats language models and rule-based approaches side by side and gives sustained attention to requirements, evaluation, and safety rather than architecture alone ^[26]. The pre-LLM literature on hallucination, reference-based metric failure, and human evaluation design reads today as an early map of exactly the problems that LLM deployment keeps rediscovering ^[8][20][21].

See also

• Natural language processing
• Natural language understanding
• Text generation models
• Hallucination
• ROUGE score
• Large language model

References

1. ^Reiter, E. and Dale, R. "Building Applied Natural Language Generation Systems." Natural Language Engineering 3(1), 1997. doi.org/...S1351324997001502
2. ^Gatt, A. and Krahmer, E. "Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation." Journal of Artificial Intelligence Research 61, 2018 (arXiv:1703.09902). arxiv.org/...1703.09902
3. ^Goldberg, E., Driedger, N. and Kittredge, R. "Using Natural-Language Processing to Produce Weather Forecasts." IEEE Expert 9(2), 1994. ieeexplore.ieee.org/...294135
4. ^Reiter, E., Sripada, S., Hunter, J., Yu, J. and Davy, I. "Choosing words in computer-generated weather forecasts." Artificial Intelligence 167, 2005. doi.org/...j.artint.2005.06.006
5. ^Gatt, A. and Reiter, E. "SimpleNLG: A Realisation Engine for Practical Applications." Proceedings of ENLG 2009. aclanthology.org/W09-0613
6. ^Papineni, K., Roukos, S., Ward, T. and Zhu, W.-J. "BLEU: a Method for Automatic Evaluation of Machine Translation." Proceedings of ACL 2002. aclanthology.org/P02-1040
7. ^Lin, C.-Y. "ROUGE: A Package for Automatic Evaluation of Summaries." Text Summarization Branches Out, 2004. aclanthology.org/W04-1013
8. ^Novikova, J., Dušek, O., Cercas Curry, A. and Rieser, V. "Why We Need New Evaluation Metrics for NLG." Proceedings of EMNLP 2017. arxiv.org/...1707.06875
9. ^Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. and Artzi, Y. "BERTScore: Evaluating Text Generation with BERT." ICLR 2020. arxiv.org/...1904.09675
10. ^Sutskever, I., Vinyals, O. and Le, Q. V. "Sequence to Sequence Learning with Neural Networks." NeurIPS 2014. arxiv.org/...1409.3215
11. ^Novikova, J., Dušek, O. and Rieser, V. "The E2E Dataset: New Challenges For End-to-End Generation." Proceedings of SIGDIAL 2017. arxiv.org/...1706.09254
12. ^Dušek, O., Novikova, J. and Rieser, V. "Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge." Computer Speech and Language, 2019 (arXiv:1901.07931). arxiv.org/...1901.07931
13. ^Gardent, C., Shimorina, A., Narayan, S. and Perez-Beltrachini, L. "The WebNLG Challenge: Generating Text from RDF Data." Proceedings of INLG 2017. aclanthology.org/W17-3518
14. ^Wiseman, S., Shieber, S. M. and Rush, A. M. "Challenges in Data-to-Document Generation." Proceedings of EMNLP 2017. arxiv.org/...1707.08052
15. ^Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019. cdn.openai.com/...upervised_multitask_learners.pdf
16. ^Brown, T. B. et al. "Language Models are Few-Shot Learners." NeurIPS 2020 (arXiv:2005.14165). arxiv.org/...2005.14165
17. ^OpenAI. "Introducing ChatGPT." November 30, 2022. openai.com/...chatgpt
18. ^Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C. and Socher, R. "CTRL: A Conditional Transformer Language Model for Controllable Generation." Salesforce Research, 2019. arxiv.org/...1909.05858
19. ^Dathathri, S. et al. "Plug and Play Language Models: A Simple Approach to Controlled Text Generation." ICLR 2020 (arXiv:1912.02164). arxiv.org/...1912.02164
20. ^Maynez, J., Narayan, S., Bohnet, B. and McDonald, R. "On Faithfulness and Factuality in Abstractive Summarization." Proceedings of ACL 2020. arxiv.org/...2005.00661
21. ^Ji, Z. et al. "Survey of Hallucination in Natural Language Generation." ACM Computing Surveys 55(12), 2023 (arXiv:2202.03629). arxiv.org/...2202.03629
22. ^Kasner, Z. and Dušek, O. "Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation." Proceedings of ACL 2024. arxiv.org/...2401.10186
23. ^Wikipedia. "Automated Insights." en.wikipedia.org/...Automated_Insights
24. ^Wikipedia. "Narrative Science." en.wikipedia.org/...Narrative_Science
25. ^Arria NLG. Company website. arria.com
26. ^Reiter, E. "Natural Language Generation." Springer, 2024. link.springer.com/...978-3-031-68582-8
27. ^Stats Perform. "Automated Insights." statsperform.com/automated-insights
28. ^Wikipedia. "Automated journalism." en.wikipedia.org/...Automated_journalism
29. ^Oremus, W. "The First News Report on the L.A. Earthquake Was Written by a Robot." Slate, March 17, 2014. slate.com/...alist-writes-article-on-la-earthquake