Natural Language Processing

Natural language processing (NLP) is the study and development of computational methods for analyzing, representing, retrieving, transforming, and generating human language. It draws on artificial intelligence, machine learning, computer science, linguistics, statistics, and related fields. NLP systems work with written language and with transcripts or symbolic representations of speech. Processing the acoustic signal itself is usually treated as speech recognition or speech synthesis, although those fields are closely connected to NLP.^[1]

NLP is an umbrella field rather than a single model or task. It includes rule-based programs, probabilistic models, neural networks, retrieval systems, and combinations of these methods. Natural language understanding concerns the analysis and interpretation of language, while natural language generation concerns the production of language. A language model estimates or assigns probabilities to sequences or other linguistic units. Large language models and generative AI overlap with NLP, but neither term is a synonym for the field as a whole.^[1]

The appropriate method depends on the task, available data, languages, domain, error costs, latency, and need for explanation or control. Pretrained Transformer models have changed many research and engineering workflows, but task-specific classifiers, rules, dictionaries, finite-state methods, and statistical sequence models remain useful where their assumptions fit the problem.^[1]

Scope and linguistic levels

Human language conveys information at several interacting levels. Morphology concerns the internal structure of words. Syntax concerns how words and phrases combine. Semantics concerns conventional meaning, while pragmatics concerns meaning in context and in relation to speakers' goals. Discourse analysis considers connections across sentences and turns. A system may need information from several levels at once. Resolving the pronoun in "The trophy did not fit in the suitcase because it was too large," for example, cannot be reduced to matching a nearby noun.^[1]

The boundary between NLP and computational linguistics is not sharp. Both investigate computational accounts of language. "NLP" often emphasizes methods and applications, while "computational linguistics" may place more emphasis on language as a scientific object, but researchers and publications commonly span both. NLP also overlaps with information retrieval, which is primarily concerned with finding relevant information, and with speech technology, which maps between audio and linguistic representations.^[1]

Several terms describe narrower activities within the field:

Scroll sideways for more →

These divisions are practical, not absolute. A question-answering system can use retrieval, entity recognition, parsing, and generation in one pipeline. A translation system both analyzes a source and generates a target. End-to-end neural models can learn several operations jointly, but the distinctions remain useful for designing data, evaluation, and safeguards.^[1]

Historical development

Early work treated language processing largely as the application of hand-written dictionaries, grammars, and transformation rules. A public Georgetown University and IBM demonstration in January 1954 translated a restricted set of Russian sentences into English. The experiment used a vocabulary of about 250 words and six grammar rules, yet it prompted broad expectations about near-term automatic translation.^[2]

The 1966 report Language and Machines, prepared by the Automatic Language Processing Advisory Committee for the U.S. National Research Council, was critical of the practical progress and economics of machine translation at that time. Its recommendations included support for basic research in computational linguistics and for machine aids that could improve human translation. The report should not be read as proving that machine translation was impossible, nor did research cease after its publication.^[3]

Also in 1966, Joseph Weizenbaum described ELIZA, a program that analyzed input through keyword-triggered decomposition rules and generated responses with associated reassembly rules. Its best-known script imitated a nondirective psychotherapist. ELIZA demonstrated how constrained pattern matching could sustain the appearance of a conversation without a general representation of the world or the user's meaning.^[4]

From the late 1980s through the 2000s, larger machine-readable corpora and increased computing power supported a shift toward statistical methods. Researchers estimated probabilities from observed data instead of specifying every decision in a grammar. In statistical machine translation, for example, Brown and colleagues formalized models that related words and alignments in bilingual sentence pairs.^[5] N-gram models estimated a token from a limited preceding context. Hidden Markov models and later conditional random fields were widely used for sequence labeling. Rule-based and statistical components were often combined rather than deployed as mutually exclusive alternatives.^[1]

Neural methods became increasingly influential during the 2010s. The word2vec work of Mikolov and colleagues introduced efficient architectures for learning continuous word embeddings from large corpora.^[6] Unlike one-hot identifiers, dense vectors can encode distributional similarities. Static embeddings still assign one vector per vocabulary item, however, so the same representation is used for a word across different contexts.

Sequence-to-sequence work in 2014 used one long short-term memory network to encode a source sequence and another to decode a target sequence, with machine translation as the main experiment.^[7] Bahdanau, Cho, and Bengio then proposed allowing a decoder to attend to different source positions as it predicted each target word, addressing the fixed-vector bottleneck in early encoder-decoder systems.^[8] These developments built on recurrent neural networks, which process a sequence through recurrent state.

The 2017 Transformer architecture replaced recurrence and convolution in its original sequence-transduction design with attention mechanisms. Its initial evaluation focused on machine translation, and its parallelizable training was an important practical property.^[9] The architecture then became a foundation for several forms of pretrained language model.

ELMo, published in 2018, produced contextual word representations from the internal states of a bidirectional language model and improved results on six reported NLP tasks.^[10] BERT, published in 2019, pretrained deep bidirectional representations with a masked-token objective and then adapted the model to tasks such as question answering and natural language inference.^[11] T5 framed a wide range of text tasks in a common text-to-text format and systematically compared transfer-learning choices.^[12] GPT-3, published in 2020, demonstrated few-shot task specification through text prompts without gradient updates at use time, while also documenting tasks on which the approach struggled and methodological concerns associated with large web corpora.^[13]

These papers mark changes in common research practice, not a clean replacement of all earlier methods. Modern systems may use pretrained encoders, autoregressive decoders, encoder-decoder models, retrieval, constraints, rules, or smaller supervised models. The history of NLP is therefore better understood as an expanding set of methods than as a single sequence in which each new architecture renders the previous one obsolete.^[1]

Core tasks

Text preparation and representation

Tokenization divides text into units such as words, subwords, or characters. The choice affects vocabulary size, sequence length, handling of spelling variation, and coverage across writing systems. Normalization may standardize selected character forms or whitespace, but indiscriminate normalization can remove distinctions that matter to the task or language. Sentence segmentation, language identification, and morphological analysis are other common preparation steps.^[1]

Representations range from sparse counts to dense vectors. Bag-of-words and term-frequency features discard word order but can remain effective for classification and retrieval. Static embeddings such as word2vec map a vocabulary item to a learned vector. Contextual models produce a representation conditioned on the surrounding sequence, so different uses of the same written word can receive different vectors.^[6][10]

Sequence and structure analysis

Part-of-speech tagging assigns grammatical categories to tokens. Named entity recognition identifies spans that refer to categories such as people, organizations, or locations, with the exact label set determined by the dataset and application. Syntactic parsing constructs constituency or dependency structures. Semantic role labeling identifies relations between predicates and their arguments. Coreference resolution groups expressions that refer to the same entity or event.^[1]

These outputs are not always ends in themselves. They can provide features or intermediate structures for search, information extraction, question answering, and other applications. Their usefulness depends on whether the labels and annotation conventions match the intended domain.

Classification and inference

Text classification assigns one or more labels to a document, sentence, or span. Common examples include topic classification, intent detection, toxicity detection, and sentiment analysis. Natural language inference classifies the relationship between a premise and a hypothesis, commonly as entailment, contradiction, or neutral. Similar-looking classification tasks may embody different definitions, annotation instructions, and social assumptions, so a label's meaning must be read from the dataset rather than inferred from its name.^[1][23]

Extraction, retrieval, and answering

Information extraction converts text into selected structured facts, such as entities, relations, events, or temporal expressions. Search systems rank documents or passages for a query. Question answering may extract a span from supplied material, select an answer from candidates, query a structured source, or generate a response. A generated answer does not by itself establish that the answer is supported by the source. Systems that combine retrieval with generation are often described as retrieval-augmented generation, and they require evaluation of both retrieval and answer behavior.^[1]

Translation, summarization, and generation

Machine translation maps content from a source language to a target language. Text summarization produces a shorter representation of a source, either by selecting source material or generating new wording. Dialogue systems choose or generate turns in an interaction. Other generation tasks include paraphrasing, grammatical error correction, data-to-text generation, and controlled generation under requirements for style or content.^[1]

Fluent output is not necessarily correct, faithful, or useful. The required standard differs by application: a creative completion can permit novelty, while a medical summary or answer grounded in a document may require every material statement to be traceable to evidence.

Methods and representations

Rules and linguistic resources

Rule-based systems encode patterns, grammars, lexicons, ontologies, or domain constraints directly. They can be transparent and precise within a narrow scope, and they can operate with little labeled data. Their disadvantages include the labor of construction, difficulty covering open-ended variation, and brittleness when input differs from anticipated patterns. Rules are still useful for deterministic formatting, terminology constraints, validation, and high-precision components in larger systems.^[1]

Statistical learning

Statistical NLP estimates regularities from corpora. A supervised classifier learns from labeled examples; an unsupervised or self-supervised method derives training signals from the data itself. Probabilistic sequence models can represent dependencies between neighboring labels and observations. Feature engineering historically supplied counts, lexical properties, part-of-speech information, and other signals to linear or probabilistic models.^[1]

Statistical performance depends on how data were sampled and labeled. If training and test examples share incidental cues, a system may score well without learning the intended relation. In a study of two natural language inference datasets, a model that saw only the hypothesis, not the premise needed for the stated task, predicted labels substantially above chance. The authors traced part of this behavior to annotation artifacts.^[18]

Neural networks and pretraining

Neural NLP learns distributed representations and nonlinear transformations. Convolutional and recurrent networks were widely used for classification and sequence processing. Attention lets a model combine information from selected positions, while Transformers use attention as the central mechanism for contextualizing sequences.^[8][9]

Pretraining first optimizes a model on a broad objective and corpus. Fine tuning then updates some or all parameters for a downstream task, although frozen representations, adapters, prompting, and other adaptation strategies are also used. Encoder-focused models are commonly suited to representation and classification, decoder-only autoregressive models to continuation and generation, and encoder-decoder models to conditional sequence transformation. These are tendencies rather than strict limits.^[1][11][12]

Prompt-based few-shot or in-context learning supplies instructions and examples in the input instead of updating weights for every task. The GPT-3 experiments established that this approach could be competitive on some evaluated datasets, but the same paper reported uneven results and limitations. It does not follow that one model performs every NLP task reliably or that task-specific evaluation is unnecessary.^[13]

Hybrid and constrained systems

Practical systems often combine methods. A pipeline can retrieve candidate passages, apply rules to enforce a schema, use a classifier to route requests, and call a generative model only for selected steps. A knowledge base can constrain entity linking. A decoder can be restricted to valid formats. Human review can be required when confidence is low or consequences are high. Hybrid design is not a distinct historical era; it is a way to assign each component a bounded responsibility.

Data and training

NLP data include raw corpora, parallel translations, transcribed speech, dictionaries, treebanks, labeled examples, preference judgments, and interaction logs. Their collection conditions affect what a system can learn and where results can be generalized. Relevant details include language varieties, speaker or author populations, genre, date range, annotation process, licensing, consent, and the intended unit of analysis.^[19]

Supervised datasets use human or programmatically assigned labels. Annotation guidelines turn an abstract task into operational categories, so disagreements can reflect ambiguity in the guidelines as well as annotator error. Inter-annotator agreement describes consistency under a particular scheme; it does not prove that the scheme is valid for every use. Crowdsourced protocols can also introduce shortcuts, as the natural language inference artifact study showed.^[18]

Self-supervised objectives create targets from otherwise unlabeled data. Autoregressive language modeling predicts subsequent tokens. Masked language modeling reconstructs selected hidden tokens. Sequence-to-sequence denoising reconstructs or transforms corrupted input. These objectives can produce reusable representations, but the training corpus still determines language coverage, domain exposure, duplicated material, and many learned associations.^[1][11][12]

Documenting data can make limitations easier to inspect. Bender and Friedman proposed data statements that record the populations and contexts represented in NLP datasets, partly to support more precise claims about generalization.^[19] Documentation does not eliminate errors or inequities, but it provides information needed to evaluate them.

Evaluation

NLP evaluation must match the system's task and intended use. Classification commonly uses accuracy, precision, recall, and the F1 score. Accuracy can conceal poor performance on rare classes. A single F1 value also depends on how class-level results are averaged. Confusion matrices and subgroup results can show failure patterns hidden by an aggregate score.^[1]

Perplexity evaluates probabilistic prediction under a particular tokenization and test distribution. It is not a universal measure of writing quality or factuality, and values from different vocabularies or preprocessing pipelines may not be directly comparable.^[1]

Generation metrics usually compare a system output with one or more reference texts. BLEU was introduced for corpus-level machine translation evaluation and combines modified n-gram precision with a brevity penalty.^[14] ROUGE is a family of recall-oriented overlap measures introduced for summary evaluation.^[15] Both can support reproducible comparisons under specified settings, but lexical overlap does not fully measure meaning, usefulness, factual support, or the validity of alternative wording. Human evaluation and task-specific checks remain important.

Multi-task suites aggregate results across datasets. The GLUE benchmark combined nine natural language understanding tasks with a diagnostic set.^[16] Rapid progress on GLUE motivated SuperGLUE, which introduced a more difficult collection and a public evaluation framework.^[17] Benchmark scores describe performance on defined datasets and metrics, not unrestricted language understanding.

Evaluation can fail in several ways. Test examples may contain shortcuts, overlap with training data, or poorly specified labels. A model can improve the average while regressing on a critical subgroup. Held-out accuracy can also miss basic behavioral failures. CheckList, a behavioral testing methodology demonstrated on three NLP tasks, found failures in both commercial and research systems and illustrates why targeted capability tests can complement ordinary test sets.^[21]

For deployment, evaluation should reflect the actual language varieties, domains, input lengths, operating conditions, and costs of error. It should include adversarial or stress testing where appropriate, uncertainty and abstention behavior, latency and resource limits, and monitoring for distribution changes. High-stakes decisions require domain review and should not be justified by a generic benchmark alone.

Applications

NLP supports search and document ranking, translation, transcription pipelines, accessibility tools, customer-service routing, writing assistance, content moderation, and extraction from legal, scientific, financial, or clinical documents. It can help organize large text collections, identify candidate records for review, or provide interfaces to structured systems.^[1]

Applications differ sharply in risk. A typo corrector and a tool that drafts a clinical note both process language, but the consequences of an error are not comparable. In lower-risk settings, convenience and latency may dominate design. In higher-risk settings, provenance, access control, auditability, calibrated uncertainty, and qualified human review become central requirements. The presence of natural-sounding text should never substitute for validation against the underlying evidence.

NLP also supports linguistic research by enabling corpus search, annotation assistance, quantitative analysis, and tests of hypotheses about language. Conversely, linguistic knowledge can guide dataset design, error analysis, and model constraints. The field is therefore both an engineering area and a means of studying language computationally.^[1]

Limitations and responsible use

Ambiguity, context, and distribution shift

Language is variable and context-dependent. The same surface form can express different meanings, and the intended interpretation may depend on shared knowledge, discourse history, social setting, or events outside the text. Systems trained in one domain may fail on another because vocabulary, genres, label frequencies, and communicative conventions change. Robust performance therefore requires more than a randomly held-out split from the same collection.^[1][21]

Language coverage

NLP resources and research attention are distributed unevenly. A 2020 study of linguistic diversity in the field found that only a small fraction of the world's more than 7,000 languages were represented in rapidly developing language technologies and conference research. The authors documented substantial differences in resources and questioned broad claims that systems were language-agnostic.^[22] Performance reported for English, or for a few high-resource languages, should not be generalized to other languages without direct evaluation.

Social bias and harmful classification

Language data reflect social relations, institutional practices, and historical inequities. Models can reproduce stereotypes or perform differently across communities and language varieties. "Bias" is not a self-defining scalar property: a responsible analysis must identify what behavior is harmful, to whom, in what context, and according to which normative standard. A survey of 146 NLP papers found that motivations and definitions were often vague or mismatched to the chosen measurements.^[23]

Factuality and provenance

Generative models can produce statements not supported by their input, commonly described as hallucination. In a human evaluation of several neural abstractive summarization systems, Maynez and colleagues found substantial unfaithful content in the outputs of every tested system, although pretrained models performed better than earlier alternatives in that study.^[24] The result is specific to the evaluated models and datasets, but it demonstrates why fluency and overlap metrics cannot establish factual support.

Retrieval, citations, constrained decoding, and post-generation verification can reduce some risks, but none guarantees correctness by itself. A retrieval system can return irrelevant evidence, a model can misstate a retrieved passage, and a citation can fail to support the sentence attached to it. Evaluation must check the complete chain from source selection to final claim.

Privacy and memorization

Training or operating on sensitive text can expose personal information through logs, outputs, model memorization, or unauthorized access. Carlini and colleagues demonstrated an extraction attack against GPT-2 that recovered hundreds of verbatim training sequences, including public personally identifiable information. This result concerns a specific model and attack, not every NLP system, but it establishes that public-source training data are not automatically safe from memorization and extraction.^[25] Data minimization, access control, retention limits, redaction, and model-specific privacy testing may be necessary.

Resources and environmental cost

The cost of NLP ranges from lightweight rules and linear classifiers to training and serving large neural networks. A 2019 study quantified financial and estimated energy costs for selected deep NLP experiments and argued that large compute requirements affect environmental impact and who can participate in research.^[26] Exact emissions depend on hardware, utilization, data-center efficiency, energy source, experiment design, and system lifetime, so old single-number estimates should not be treated as universal current values.

Documentation and oversight

Model cards were proposed as concise documents that state intended uses, evaluation procedures, performance across relevant conditions, and other limitations of trained models.^[20] Together with data documentation, reproducible evaluation, and incident monitoring, such records can clarify where evidence supports a deployment and where it does not.

Responsible NLP practice sets requirements from the application rather than from model popularity. It defines unacceptable errors, evaluates representative users and language varieties, protects data, records provenance, provides ways to contest or correct outputs, and assigns meaningful human authority where consequences warrant it. No single metric, benchmark, or architecture resolves these governance questions.

References

1. ^Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models, third-edition online manuscript, January 6, 2026. web.stanford.edu/...slp3
2. ^W. John Hutchins. "The Georgetown-IBM Experiment Demonstrated in January 1954." Proceedings of the 6th Conference of the Association for Machine Translation in the Americas, 2004. aclanthology.org/2004.amta-papers.12
3. ^National Research Council, Automatic Language Processing Advisory Committee. Language and Machines: Computers in Translation and Linguistics, 1966. nationalacademies.org/...20813
4. ^Joseph Weizenbaum. "ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine." Communications of the ACM 9(1), 1966. doi.org/...365153.365168
5. ^Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. "The Mathematics of Statistical Machine Translation: Parameter Estimation." Computational Linguistics 19(2), 1993. aclanthology.org/J93-2003
6. ^Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. "Efficient Estimation of Word Representations in Vector Space." arXiv, 2013. arxiv.org/...1301.3781
7. ^Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. "Sequence to Sequence Learning with Neural Networks." Advances in Neural Information Processing Systems 27, 2014. papers.nips.cc/...bf9f257297f410bb7eca942-Abstract
8. ^Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. "Neural Machine Translation by Jointly Learning to Align and Translate." ICLR 2015. arxiv.org/...1409.0473
9. ^Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. "Attention Is All You Need." Advances in Neural Information Processing Systems 30, 2017. papers.nips.cc/...47dee91fbd053c1c4a845aa-Abstract
10. ^Matthew E. Peters, Mark Neumann, Mohit Iyyer, et al. "Deep Contextualized Word Representations." Proceedings of NAACL-HLT, 2018. aclanthology.org/N18-1202
11. ^Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of NAACL-HLT, 2019. aclanthology.org/N19-1423
12. ^Colin Raffel, Noam Shazeer, Adam Roberts, et al. "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." Journal of Machine Learning Research 21(140), 2020. jmlr.org/...20-074
13. ^Tom B. Brown, Benjamin Mann, Nick Ryder, et al. "Language Models Are Few-Shot Learners." Advances in Neural Information Processing Systems 33, 2020. papers.nips.cc/...fcb4967418bfb8ac142f64a-Abstract
14. ^Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. "BLEU: A Method for Automatic Evaluation of Machine Translation." Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002. aclanthology.org/P02-1040
15. ^Chin-Yew Lin. "ROUGE: A Package for Automatic Evaluation of Summaries." Text Summarization Branches Out, 2004. aclanthology.org/W04-1013
16. ^Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding." Proceedings of the 2018 EMNLP Workshop BlackboxNLP, 2018. aclanthology.org/W18-5446
17. ^Alex Wang, Yada Pruksachatkun, Nikita Nangia, et al. "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems." Advances in Neural Information Processing Systems 32, 2019. papers.nips.cc/...fe7fab6f046bf4923da8de6-Abstract
18. ^Suchin Gururangan, Swabha Swayamdipta, Omer Levy, et al. "Annotation Artifacts in Natural Language Inference Data." Proceedings of NAACL-HLT, 2018. aclanthology.org/N18-2017
19. ^Emily M. Bender and Batya Friedman. "Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science." Transactions of the Association for Computational Linguistics 6, 2018. aclanthology.org/Q18-1041
20. ^Margaret Mitchell, Simone Wu, Andrew Zaldivar, et al. "Model Cards for Model Reporting." Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019. arxiv.org/...1810.03993
21. ^Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. aclanthology.org/2020.acl-main.442
22. ^Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. "The State and Fate of Linguistic Diversity and Inclusion in the NLP World." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. aclanthology.org/2020.acl-main.560
23. ^Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. "Language (Technology) Is Power: A Critical Survey of 'Bias' in NLP." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. aclanthology.org/2020.acl-main.485
24. ^Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. "On Faithfulness and Factuality in Abstractive Summarization." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. aclanthology.org/2020.acl-main.173
25. ^Nicholas Carlini, Florian Tramèr, Eric Wallace, et al. "Extracting Training Data from Large Language Models." 30th USENIX Security Symposium, 2021. usenix.org/...carlini-extracting
26. ^Emma Strubell, Ananya Ganesh, and Andrew McCallum. "Energy and Policy Considerations for Deep Learning in NLP." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. aclanthology.org/P19-1355