Last reviewed
May 17, 2026
Sources
No citations yet
Review status
Needs citations
Revision
v2 · 4,516 words
Add missing citations, update stale details, or suggest a clearer explanation.
GloVe, short for Global Vectors for Word Representation, is an unsupervised learning algorithm for producing dense vector representations of words. It was introduced by Jeffrey Pennington, Richard Socher, and [[christopher_manning|Christopher D. Manning]] of the [[stanford_nlp|Stanford NLP Group]] in a 2014 paper presented at the Empirical Methods in Natural Language Processing (EMNLP) conference in Doha, Qatar.[^1] GloVe maps each word in a vocabulary to a single fixed vector such that the geometric relationships between vectors reflect the statistical regularities of how words co-occur across a large text corpus. It is one of the canonical static [[word_embedding|word embedding]] methods of the mid-2010s and, alongside [[word2vec|Word2Vec]], it played a central role in the deep learning revolution in natural language processing that preceded the transformer era.[^2]
GloVe was designed to combine the two main families of word vector models that existed before it: global matrix factorization methods such as latent semantic analysis (LSA), which use corpus-wide statistics but perform poorly on word analogy tasks, and local context window methods such as Word2Vec, which capture analogy structure well but ignore global co-occurrence counts.[^1] The Stanford team showed that a simple weighted least-squares regression on the logarithm of word-word co-occurrence counts could keep the strengths of both approaches while shedding most of their weaknesses. The resulting vectors achieved a then state-of-the-art 75 percent accuracy on a popular word analogy benchmark and produced linear substructures such as the famous relation vec("king") - vec("man") + vec("woman") ≈ vec("queen").[^1]
Along with the paper, the Stanford NLP Group released training code and a set of pre-trained vectors on Wikipedia, Gigaword, Common Crawl, and Twitter corpora at dimensions of 50, 100, 200, and 300.[^2] These files were downloaded millions of times and became a default initialization for neural NLP models from 2014 through roughly 2018. Although contextual [[embeddings|embeddings]] such as [[elmo|ELMo]] and [[bert|BERT]], built on recurrent and transformer architectures, eventually displaced GloVe as the front-line representation for production systems, GloVe vectors remain in active use as a lightweight baseline, as initialization for smaller models, and in linguistic and bias research.[^10][^11]
By 2013, two distinct lines of research dominated distributional word representations. The first was global matrix factorization: methods such as latent semantic analysis (LSA), Hellinger PCA, and the Hyperspace Analogue to Language (HAL) constructed a large matrix of word counts, then decomposed it via singular value decomposition or a related factorization. These methods used corpus statistics efficiently but performed weakly on word analogy tasks.[^1]
The second line was the local context window family. The breakthrough came in 2013 with Tomas Mikolov's [[word2vec|Word2Vec]] papers from Google.[^4][^5] Word2Vec introduced two shallow neural architectures, Continuous Bag of Words (CBOW) and Skip-gram, that scanned a corpus with a sliding window and learned vectors by predicting words from their immediate neighbors. Word2Vec vectors were strikingly good at analogy tasks and trained at unprecedented speed, but the original formulation discarded the global co-occurrence counts of the corpus and processed each window independently.
GloVe was conceived to bridge these two traditions. Pennington, Socher, and Manning argued that the key information for distributional semantics lives in the ratios of co-occurrence probabilities, and that those ratios can be made linear in vector space by predicting the logarithm of co-occurrence counts directly with a simple bilinear model.[^1] The result was a count-based model that inherited the vector arithmetic and analogy structure that had made Word2Vec famous, while reusing corpus-level statistics in a single optimization pass rather than reprocessing the entire corpus by streaming windows.
GloVe begins with a single global pass over the corpus to build a word-word co-occurrence matrix X. Each entry X_ij records how often word j appears in the context of word i, where context is defined by a symmetric window of fixed size (typically 10 tokens). Co-occurrences within a window are weighted by 1/d, where d is the distance between the two words, so that nearby words contribute more than distant ones.[^1] The row sum X_i = Σ_k X_ik gives the total context count for word i, and the conditional probability is P_ij = X_ij / X_i.
For a moderately sized corpus the matrix is enormous in dimensionality (vocabulary squared) but extremely sparse in entries, since most word pairs never co-occur within a window. GloVe exploits this sparsity by training only on the nonzero entries. The cost of constructing the matrix scales with the size of the corpus, but only once.
The central insight of the paper is that the ratio P_ik / P_jk carries more discriminative information than either probability alone. To illustrate, the authors compare the words ice and steam against probe words solid, gas, water, and fashion. The ratio P(solid|ice) / P(solid|steam) is large because solid is much more associated with ice, the ratio for gas is small for the symmetric reason, and the ratio is close to one for both water (related to both) and fashion (related to neither).[^1] The ratio therefore filters out content that is uninformative for distinguishing the two target words, while amplifying contrasts that are diagnostic of meaning.
GloVe parametrizes word vectors w_i and separate context vectors õ_j so that their dot product approximates the log of the co-occurrence count, with bias terms b_i and ù_j absorbing word-specific frequencies:[^1]
w_i^T õ_j + b_i + ù_j ≈ log(X_ij)
This form is derived in the paper through a chain of arguments that begins by requiring vector differences to be linear in the log probability ratios, then enforcing symmetry between target and context words. The two vector sets w and õ are equivalent in expectation but are initialized separately and trained independently, which acts as a form of regularization.
The full training objective is a weighted least-squares regression over all nonzero entries of the co-occurrence matrix:[^1]
J = Σ_{i,j} f(X_ij) (w_i^T õ_j + b_i + ù_j - log X_ij)^2
The weighting function f is what makes the model behave well in practice. A naive squared loss would treat all co-occurrences equally, letting extremely common pairs such as the-of dominate training while giving as much weight to noisy rare co-occurrences as to informative frequent ones. The chosen function caps the influence of high counts and gently dampens low ones:
f(x) = (x / x_max)^α if x < x_max, and f(x) = 1 otherwise.
The original paper used x_max = 100 and α = 3/4, the same exponent that appeared in the Word2Vec negative sampling distribution.[^1][^5] The function is zero at x = 0, which lets the model skip the vast number of zero entries in X and operate only on the nonzero counts. This combination of corpus-wide statistics with a sparse, count-only training signal is what gives GloVe its efficiency advantage over naive matrix factorization, which would require eigendecomposing the entire dense matrix.
The paper devotes considerable space to relating GloVe to the Skip-gram with negative sampling objective. Pennington et al. show that Skip-gram's softmax-with-negative-sampling cost can, under simplifying assumptions, be cast in a form whose global objective is structurally similar to GloVe's: a weighted regression on a transformation of co-occurrence counts.[^1] The difference lies in the weighting function and the specific transformation. GloVe's explicit choice of log counts and the (x/x_max)^α weight, the authors argue, is what makes the global statistics tractable and the analogy substructure clean. The connection was later formalized in greater detail by Levy and Goldberg.[^7]
Training is performed with the AdaGrad optimizer at an initial learning rate of 0.05, with batches of randomly sampled nonzero entries. The original paper trained for 50 iterations for vector dimensionalities below 300 and 100 iterations otherwise.[^1] After training, the final word vector for each word is taken to be the sum w_i + õ_i, which gave a small accuracy boost over either set on its own. Each word ends up with a single dense vector, typically of dimension 50, 100, 200, or 300. Unlike contextual models such as [[bert|BERT]], the vector for a given surface word does not change with context.
The Stanford NLP Group released a series of pre-trained vector files alongside the original paper, and has updated and extended these releases through subsequent years.[^2] The downloads are distributed as plain text files in a simple line-based format that has become a de facto interchange standard for static word embeddings: each line contains a word followed by d space-separated floating point values.
| Corpus | Tokens | Vocabulary | Dimensions | File size |
|---|---|---|---|---|
| Wikipedia 2014 + Gigaword 5 | 6 billion | 400,000 | 50, 100, 200, 300 | 822 MB |
| Common Crawl (uncased) | 42 billion | 1.9 million | 300 | 1.75 GB |
| Common Crawl (cased) | 840 billion | 2.2 million | 300 | 2.03 GB |
| 27 billion | 1.2 million | 25, 50, 100, 200 | 1.42 GB |
The Wikipedia plus Gigaword bundle was the most widely used package, since it provided four different dimensionalities in a single download and covered most of the formal English vocabulary that downstream NLP tasks of the period cared about.[^2] The Twitter set, with its 25- and 50-dimensional options and informal vocabulary, was popular for social media tasks where slang, hashtags, and emoji-adjacent tokens dominated. The 840-billion-token cased Common Crawl release, with a 2.2-million-word vocabulary, remained the largest English static word embedding distribution for years and was widely cited as the most comprehensive single-vector representation of English from the 2010s.
In 2024 the Stanford NLP Group refreshed and expanded the original distribution. The new releases include vectors trained on a 220-billion-token slice of the Dolma corpus and an updated Wikipedia plus Gigaword 5 set with 11.9 billion tokens and a 1.2-million-word vocabulary across four dimensions.[^2] The release was documented in a 2025 arXiv report by Riley Carlson, John Bauer, and Christopher D. Manning titled A New Pair of GloVes, which describes the data sources, preprocessing pipeline, and evaluation of the refreshed vectors and reports improvements on contemporary named entity recognition benchmarks while preserving performance on classical analogy and similarity tasks.[^13]
| Release | Tokens | Vocabulary | Dimensions | File size |
|---|---|---|---|---|
| Dolma 2024 | 220 billion | 1.2 million | 300 | 1.6 GB |
| Wikipedia + Gigaword 5 (2024) | 11.9 billion | 1.2 million | 50 | 290 MB |
| Wikipedia + Gigaword 5 (2024) | 11.9 billion | 1.2 million | 100 | 560 MB |
| Wikipedia + Gigaword 5 (2024) | 11.9 billion | 1.2 million | 200 | 1.1 GB |
| Wikipedia + Gigaword 5 (2024) | 11.9 billion | 1.2 million | 300 | 1.6 GB |
The continued release of pre-trained vectors more than a decade after the original paper reflects the enduring popularity of GloVe as a fast, deterministic, and reproducible baseline for word-level NLP work even in the era of large language models.
GloVe is best understood next to its two closest peers in the static word embedding family: [[word2vec|Word2Vec]], released by Mikolov and colleagues at Google in 2013, and [[fasttext|FastText]], released by Bojanowski, Grave, Joulin, and Mikolov at Facebook AI Research in 2016 and 2017.[^4][^6] All three produce a single fixed vector per word and rely on the distributional hypothesis that words in similar contexts have similar meanings, but they differ in how they ingest the corpus and what units they treat as primitive.
| Property | GloVe | Word2Vec | FastText |
|---|---|---|---|
| Year | 2014 | 2013 | 2016 to 2017 |
| Authors | Pennington, Socher, Manning (Stanford) | Mikolov et al. (Google) | Bojanowski, Grave, Joulin, Mikolov (Facebook AI Research) |
| Statistics used | Global word-word co-occurrence counts | Local sliding windows | Local windows over character n-grams |
| Training objective | Weighted least squares on log co-occurrence | Predict context (Skip-gram) or predict word from context (CBOW) with negative sampling or hierarchical softmax | Same as Word2Vec but per character n-gram, words are sums of n-gram vectors |
| Subword information | None | None | Yes, character n-grams of length 3 to 6 |
| Out-of-vocabulary words | Cannot represent | Cannot represent | Composes a vector from constituent n-grams |
| Memory at training time | High, requires co-occurrence matrix | Low, online streaming | Moderate to high, larger model with n-gram vocabulary |
| Strength | Semantic analogies, smooth use of corpus statistics, robust on small corpora when pre-trained on a large one | Fast, strong on syntactic analogies, simple to train | Morphology-rich languages, OOV handling, named entity recognition |
| Standard pre-trained release | 50d, 100d, 200d, 300d on Wikipedia plus Gigaword, Common Crawl, Twitter | 300d on Google News (100B tokens) | 300d for 157 languages on Wikipedia and Common Crawl |
In the original Pennington et al. paper, GloVe at 300 dimensions trained on the 6-billion-token Wikipedia plus Gigaword corpus reached 71.7 percent accuracy on the standard analogy task, compared with 65.7 percent for Skip-gram and 60.1 percent for CBOW under the same setup, while a 42-billion-token Common Crawl run pushed accuracy up to 75 percent.[^1] Subsequent benchmarks have shown the gap depends heavily on hyperparameter tuning. As a rule of thumb, GloVe tends to do slightly better on semantic analogies (capital-country, currency, family relations), while Word2Vec Skip-gram tends to do slightly better on syntactic analogies. FastText, with its character n-gram backbone, frequently outperforms both on rare words, morphologically rich languages, and out-of-vocabulary tokens.[^6] Word2Vec is the most memory-efficient because it streams the corpus, while GloVe pays a one-time cost to construct the global co-occurrence matrix.
The most cited example of GloVe's geometric structure is the algebraic relationship
vec("king") - vec("man") + vec("woman") ≈ vec("queen"),
which captures the gender axis between male and female royalty in a single vector difference. The same arithmetic recovers capital-country pairs (Paris is to France as Tokyo is to Japan), comparative and superlative forms of adjectives, verb tenses, and currency-country relationships.[^1] GloVe was not the first model to display this property; Mikolov's Word2Vec papers had already demonstrated similar arithmetic.[^4][^5] But GloVe's paper provided a clean mathematical motivation for why such linear substructures should emerge: if vector dot products approximate log co-occurrence probabilities, then vector differences approximate logs of probability ratios, which is exactly what the king-queen analogy expresses.[^1] The result is sometimes overstated in popular accounts; strict implementations exclude the input words from the candidate set when searching for the nearest neighbor, and the recovered vector is rarely exactly equal to queen but rather closest in cosine similarity. Critical follow-up work by Linzen and others has scrutinized exactly how much of the analogy success is contributed by the bias term in the cosine score versus the genuine geometric structure of the embedding space.
The original GloVe paper introduced or popularized several evaluation tasks that became standard for comparing static word embeddings.[^1]
The analogy task asks whether vec(a) - vec(b) + vec(c) is closest to vec(d) for question quadruples like Athens, Greece, Berlin, ?. The dataset compiled by Mikolov et al. with 19,544 questions split into semantic and syntactic items became the standard yardstick.[^4] Reported numbers in the original paper:[^1]
| Model | Vector size | Corpus size | Total accuracy | Semantic | Syntactic |
|---|---|---|---|---|---|
| ivLBL | 100 | 1.5B | 60.0 | 53.2 | 65.6 |
| HPCA | 100 | 1.6B | 4.2 | 4.9 | 3.6 |
| GloVe | 100 | 1.6B | 67.5 | 78.8 | 58.4 |
| SVD-L | 300 | 6B | 65.7 | 56.6 | 73.0 |
| CBOW | 300 | 6B | 60.1 | 16.1 | 91.4 |
| Skip-gram | 300 | 6B | 65.7 | 50.0 | 78.7 |
| GloVe | 300 | 6B | 71.7 | 77.4 | 67.0 |
| GloVe | 300 | 42B | 75.0 | 81.9 | 69.3 |
Word similarity benchmarks correlate model-predicted cosine similarities with human judgments on datasets such as WordSim-353, MC, RG, SCWS, and RareWords. GloVe vectors at 300 dimensions trained on the 42-billion Common Crawl corpus achieved Spearman correlations of 75.9 on WordSim-353, 83.6 on MC, and 82.9 on RG, beating LSA-derived baselines and matching or exceeding Skip-gram on most subsets.[^1] These numbers established GloVe as a high water mark for static distributional similarity, a position it held until contextual embeddings began routinely topping the same benchmarks in 2018.
GloVe quickly became a standard initialization for sequence-labeling models. On the CoNLL-2003 NER task using a CRF-based model with discrete features plus word vectors, 300-dimensional GloVe trained on 42B Common Crawl tokens reached an F1 of 88.3, slightly above CBOW (87.9), Skip-gram (88.3), and HPCA (88.7).[^1] Deep BiLSTM-CRF NER systems frequently used 100- or 300-dimensional GloVe vectors as input through 2017 and 2018, before being displaced by contextual representations from ELMo and BERT.[^10][^11] Beyond NER, GloVe became the default starting point for text classification, sentiment analysis, machine translation encoders, question answering, and semantic role labeling. In sentiment analysis specifically, pre-trained GloVe embeddings paired with bidirectional LSTM or CNN classifiers were the canonical baseline architecture on benchmarks such as the Stanford Sentiment Treebank and IMDB review datasets from 2014 through 2018.
GloVe and Word2Vec together inaugurated the period sometimes called the deep learning revolution in natural language processing, roughly spanning 2013 to 2017. Before this period, NLP systems were dominated by sparse, hand-crafted feature pipelines built on top of part-of-speech taggers, dependency parsers, and lexicons. After Word2Vec demonstrated the practical power of dense distributional representations and GloVe gave the field a clean mathematical framing of why vector arithmetic should work, virtually every published NLP system began with a dense word embedding layer. The pre-trained vectors became a default component of countless tutorials, workshops, and Kaggle competitions.[^2]
GloVe also normalized the practice of releasing both code and pre-trained model artifacts. The Stanford team's decision to publish multiple sizes of vectors trained on multiple corpora helped accelerate adoption and prefigured the modern open-source NLP ecosystem epitomized by Hugging Face's model hub. Citation counts for the Pennington, Socher, and Manning paper run into the tens of thousands, placing it among the most cited NLP papers of the 2010s.
The reference implementation, written in C, is hosted as the stanfordnlp/GloVe repository on GitHub and is released under the Apache License 2.0.[^3] The training pipeline is split into four command-line stages: vocab_count collects unigram counts, cooccur builds the sparse co-occurrence matrix, shuffle randomizes the order of nonzero entries to support stochastic gradient updates, and glove runs the AdaGrad training itself. Third-party implementations exist in Python (gensim, text2vec, glove-python-binary), R, and Julia. The widely used gensim library provides a KeyedVectors interface that loads any GloVe-format text file and supports the same lookup, similarity, and analogy operations as Word2Vec. The popular spaCy library used GloVe vectors as the default semantic representation in its English models for several years.
GloVe shares the fundamental limitations of every static word embedding model. The most important is the lack of context sensitivity: every occurrence of a polysemous or homographic word receives the same vector, regardless of the surrounding sentence. Bank in river bank and bank in bank account are conflated, as are the noun and verb senses of bark, book, and run. This single-vector-per-word constraint motivated the development of contextual word representations such as [[elmo|ELMo]] and the transformer-based [[bert|BERT]], which generalized the idea with self-attention and produced a different vector for every token occurrence.[^10][^11]
Out-of-vocabulary (OOV) words pose a second limitation. Any word not seen during training has no vector and must be handled by a fallback strategy such as zero vectors, random vectors, or a generic <UNK> token. This is particularly painful for morphologically rich languages with productive inflection, for technical jargon, and for proper nouns. FastText's subword n-gram approach explicitly addressed this concern by composing OOV vectors from character n-grams.[^6]
GloVe also inherits social biases present in its training corpora. Studies including Bolukbasi et al.'s 2016 paper on man-is-to-computer-programmer-as-woman-is-to-homemaker analogies and Caliskan et al.'s 2017 Word Embedding Association Test (WEAT) demonstrated that GloVe and Word2Vec vectors encode gender, racial, and occupational stereotypes with measurable cosine-similarity signatures.[^8][^9] Various debiasing procedures have been proposed, but none entirely remove the underlying bias, and the literature increasingly converged on the view that distributional embeddings will always reflect the statistical patterns of their corpora and that downstream mitigation is more reliable than upstream surgery.
The pre-trained vectors are also frozen in time, with no entries for terms that became commonplace after their training cut-off. For applications that need open-vocabulary handling, FastText's subword approach or contextual models with byte-pair or WordPiece tokenization are usually preferable. GloVe also requires a fully constructed co-occurrence matrix to begin training, which can become memory-intensive for very large corpora. Word2Vec's streaming approach side-steps this concern, part of why Word2Vec remained popular under tight infrastructure constraints. The 2024 refresh of GloVe partly addresses the staleness problem by including a Dolma-trained release that covers newer vocabulary, but no static embedding can match the dynamic representation of contextual models or large language model token embeddings.
In 2018 [[elmo|ELMo]] introduced contextual word representations on top of bidirectional language models, and later that year [[bert|BERT]] extended the idea using stacked transformer layers and masked-language-model pretraining.[^10][^11] These contextual models offered a unified solution to polysemy, syntactic ambiguity, and downstream transfer learning, and rapidly displaced GloVe as the front-line representation for production NLP. By 2020, GloVe was relegated to baseline and legacy status in benchmark papers.
Nonetheless, GloVe vectors remain widely used as a fast deterministic baseline, as initialization for small recurrent or convolutional networks where running a transformer is too expensive, and as a research tool for studying the geometry of distributional semantics, word sense, and lexical bias. The Stanford team's 2024 refresh of GloVe vectors trained on the Dolma corpus shows that interest in the model has not fully waned a decade after its introduction.[^2][^13] Carlson, Bauer, and Manning's 2025 report A New Pair of GloVes documents the data sources and preprocessing details that were absent from the original 2014 release, addressing a long-standing reproducibility gap in the original distribution.[^13]
The 2014 paper by Levy and Goldberg, Neural Word Embedding as Implicit Matrix Factorization, formally connected Word2Vec's Skip-gram with negative sampling to a shifted positive pointwise mutual information matrix factorization, which sits intellectually next to GloVe's explicit log co-occurrence factorization.[^7] Contextual models such as ELMo and BERT can be viewed as deep, contextual generalizations of the same distributional principle that GloVe captured statically. The line of work running from LSA through GloVe and Word2Vec to ELMo, BERT, and modern large language model token embeddings is a continuous arc, and GloVe occupies a clearly identifiable point on that arc: the first model to make explicit the connection between corpus-wide co-occurrence statistics, matrix factorization, and the vector arithmetic of analogy.
The GloVe code at stanfordnlp/GloVe is released under the Apache License, Version 2.0, which permits commercial and academic use with the standard requirements of attribution and notice preservation.[^3] The pre-trained vectors are released separately under the Public Domain Dedication and License (PDDL) 1.0, an Open Data Commons license that places the data files effectively in the public domain.[^2] This permissive licensing was a significant factor in GloVe's broad uptake; researchers and engineers could freely redistribute the vectors in derivative projects, embed them into commercial systems, and use them as inputs to other models without negotiating a separate license. The 2024 vector refresh was released under the same terms.
In the era of large language models, GloVe occupies a niche but stable position in the NLP toolbox. It is most useful when the application has tight latency or memory constraints that rule out running a transformer per query, when the task is fundamentally about lexical semantics rather than contextual interpretation (keyword expansion, simple search relevance, thesaurus construction), when the dataset is small and a frozen pre-trained representation is preferable to fine-tuning a large model, or when the goal is to study the geometric properties of word vectors themselves. GloVe is also frequently used in social science and digital humanities research where the unit of analysis is a word type rather than a token occurrence, and where the deterministic, reproducible nature of static embeddings is methodologically desirable.
Educational use is another lasting application. GloVe's mathematical framing is unusually clean and the training pipeline is exposed in plain command-line steps, making it a common pedagogical example for distributional semantics, matrix factorization, and the relationship between count-based and predictive embedding methods.