CtrlK
Cosine similarity is a measure of similarity between two non-zero vectors that calculates the cosine of the angle between them. It is defined as the dot product of the vectors divided by the product of their magnitudes, producing a value between -1 and 1. In artificial intelligence and machine learning, cosine similarity has become one of the most widely used similarity metrics, particularly for comparing text embeddings, measuring semantic relatedness, powering vector search in retrieval-augmented generation (RAG) systems, and defining training objectives in contrastive learning. Its popularity stems from a key property: it measures the orientation (direction) of vectors while being invariant to their magnitude, making it well-suited for comparing learned representations where direction encodes meaning.
Given two vectors A and B in n-dimensional space, cosine similarity is defined as:
cos(theta) = (A . B) / (||A|| * ||B||)
where:
Expanding the formula for two n-dimensional vectors:
cos(theta) = (A_1B_1 + A_2B_2 + ... + A_n*B_n) / (sqrt(A_1^2 + A_2^2 + ... + A_n^2) * sqrt(B_1^2 + B_2^2 + ... + B_n^2))
| Cosine Value | Angle | Interpretation |
|---|---|---|
| 1.0 | 0 degrees | Vectors point in exactly the same direction (identical orientation) |
| 0.5 to 0.9 | 26 to 60 degrees | Moderately to highly similar |
| 0.0 | 90 degrees | Vectors are orthogonal (no similarity) |
| -0.5 to -0.1 | 96 to 120 degrees | Moderately dissimilar |
| -1.0 | 180 degrees | Vectors point in exactly opposite directions |
For most applications with neural network embeddings, similarity values cluster in the positive range (roughly 0.0 to 1.0), because embedding spaces tend to use only a portion of the available directional space.
A related quantity, cosine distance, is defined as 1 - cos(theta), and ranges from 0 (identical) to 2 (opposite). Cosine distance is used when a distance metric (lower is more similar) is needed rather than a similarity metric (higher is more similar).
The most important property of cosine similarity for AI applications is its invariance to vector magnitude. Two vectors that point in the same direction have cosine similarity 1.0, regardless of their lengths. This matters because in many representation learning settings, the direction of a vector encodes semantic meaning while the magnitude may reflect irrelevant factors such as document length, word frequency, or arbitrary scale differences in the embedding model.
For example, in traditional TF-IDF document representations, a document that is twice as long as another might have TF-IDF vectors with roughly double the magnitude, even if both documents discuss the same topics. Cosine similarity correctly identifies them as similar by focusing on the proportions of terms rather than their absolute counts [1].
In high-dimensional spaces (hundreds or thousands of dimensions, as is typical for neural embeddings), Euclidean distance becomes less discriminative because all points tend to be roughly equidistant. This phenomenon, known as the "curse of dimensionality," makes absolute distance values difficult to interpret. Cosine similarity, by focusing on angular relationships rather than absolute distances, remains discriminative and interpretable in high dimensions [2].
Cosine similarity requires only a dot product and two norm computations, all of which are highly optimized on modern hardware. When vectors are pre-normalized to unit length (||A|| = ||B|| = 1), cosine similarity reduces to a simple dot product A . B, which can be computed extremely efficiently using SIMD instructions, GPU tensor cores, or specialized vector database hardware.
Cosine similarity is the standard metric for comparing vector representations (embeddings) across natural language processing, information retrieval, and recommendation systems.
The use of cosine similarity for measuring word relatedness became widespread with word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014). These models map words to dense vectors where semantic relationships are encoded as geometric relationships [3].
Classic examples of cosine similarity in word embedding spaces:
| Word Pair | Typical Cosine Similarity | Relationship |
|---|---|---|
| king, queen | ~0.75 | Semantic (gender variant) |
| cat, dog | ~0.76 | Semantic (both animals) |
| car, automobile | ~0.85 | Near-synonym |
| king, carrot | ~0.15 | Unrelated |
| good, bad | ~0.45 | Antonyms (still somewhat similar because they appear in similar contexts) |
A celebrated property of word2vec embeddings is that semantic analogies correspond to vector arithmetic. The classic example "king - man + woman is close to queen" works because cosine similarity in the embedding space captures these regular semantic relationships [3].
Modern embedding models such as Sentence-BERT (Reimers and Gurevych, 2019), E5 (Wang et al., 2022), and OpenAI's text-embedding models produce fixed-size vectors for entire sentences or documents. These models are specifically trained so that cosine similarity between embeddings reflects semantic similarity between the corresponding texts [4].
| Embedding Model | Dimensions | Typical Use | Similarity Metric |
|---|---|---|---|
| text-embedding-3-large (OpenAI) | 3072 | General-purpose text similarity | Cosine |
| E5-large-v2 | 1024 | Search and retrieval | Cosine |
| all-MiniLM-L6-v2 (SBERT) | 384 | Lightweight semantic similarity | Cosine |
| BGE-large-en-v1.5 | 1024 | Search and retrieval | Cosine |
| Cohere embed-v3 | 1024 | Multilingual search | Cosine or dot product |
These models are typically evaluated on benchmarks like MTEB (Massive Text Embedding Benchmark), where cosine similarity is used as the default similarity function for retrieval, clustering, and semantic textual similarity tasks.
Contrastive learning methods train models to produce embeddings where similar items have high cosine similarity and dissimilar items have low cosine similarity. The training objective directly optimizes cosine similarity (or equivalently, the dot product of L2-normalized vectors).
CLIP (Contrastive Language-Image Pre-training), developed by OpenAI (Radford et al., 2021), learns a shared embedding space for images and text [5]. During training, CLIP processes batches of (image, text) pairs and computes the cosine similarity between every image embedding and every text embedding in the batch. The model is trained to maximize cosine similarity for matching pairs and minimize it for non-matching pairs, using a symmetric cross-entropy loss over the similarity matrix.
At inference time, CLIP enables zero-shot image classification by computing the cosine similarity between an image embedding and text embeddings of candidate class descriptions (e.g., "a photo of a cat," "a photo of a dog"). The class with the highest cosine similarity to the image is selected as the prediction [5].
SimCLR (Chen et al., 2020) is a framework for self-supervised visual representation learning that uses cosine similarity as the core component of its NT-Xent (Normalized Temperature-scaled Cross-Entropy) loss [6]. Given a batch of images, SimCLR creates two augmented views of each image, encodes them, and projects them to a lower-dimensional space. The loss function for a positive pair (i, j) is:
loss_ij = -log(exp(sim(z_i, z_j) / tau) / sum(exp(sim(z_i, z_k) / tau) for all k != i))
where sim(z_i, z_j) = (z_i . z_j) / (||z_i|| * ||z_j||) is the cosine similarity between L2-normalized representations, and tau is a temperature parameter that controls the sharpness of the distribution.
The temperature parameter is critical: lower temperatures make the model focus more on the hardest negative examples, while higher temperatures smooth the distribution. SimCLR found tau = 0.5 to work well across many settings [6].
Cosine similarity is central to the training objectives of many other contrastive and self-supervised methods:
| Method | Domain | How Cosine Similarity Is Used |
|---|---|---|
| CLIP | Vision-language | Cross-modal matching of image and text embeddings |
| SimCLR | Vision | Self-supervised learning with augmented view pairs |
| MoCo | Vision | Momentum contrast with cosine similarity queue |
| SimCSE | NLP | Sentence embedding learning with dropout augmentation |
| ALIGN | Vision-language | Image-text alignment (similar to CLIP) |
| DINO | Vision | Self-distillation with cosine similarity |
| Barlow Twins | Vision | Cross-correlation matrix objective (related to cosine) |
Retrieval-augmented generation (RAG) systems use cosine similarity as the primary mechanism for finding relevant documents to include in a large language model's context. The process works as follows:
Vector databases like Pinecone, Weaviate, Qdrant, Milvus, and Chroma support cosine similarity as a built-in distance metric. At scale, exact cosine similarity computation with every vector becomes expensive, so these systems use approximate nearest neighbor (ANN) algorithms such as HNSW (Hierarchical Navigable Small World) and IVF (Inverted File Index) to perform fast approximate searches [7].
| Vector Database | Supported Metrics | Default Metric |
|---|---|---|
| Pinecone | Cosine, dot product, Euclidean | Cosine |
| Weaviate | Cosine, dot product, L2 | Cosine |
| Qdrant | Cosine, dot product, Euclidean | Cosine |
| Milvus | Cosine, IP, L2 | Varies |
| Chroma | Cosine, L2, IP | L2 |
| pgvector | Cosine, L2, inner product | Varies |
Semantic search more broadly relies on cosine similarity. Unlike keyword-based search (which matches exact terms), semantic search compares the meaning of queries and documents via their embeddings. A query like "how to fix a flat tire" can match a document titled "changing a punctured tire" because their embeddings point in similar directions, even though they share few words.
Cosine similarity is one of several metrics used to compare vectors. Understanding the relationships and differences between these metrics is important for choosing the right one.
The dot product (inner product) of two vectors A and B is:
A . B = sum(A_i * B_i) = ||A|| * ||B|| * cos(theta)
The dot product equals cosine similarity multiplied by the magnitudes of both vectors. When vectors are L2-normalized (unit vectors), the dot product and cosine similarity are identical. Many contrastive learning frameworks normalize embeddings before computing similarities, making the choice between dot product and cosine similarity irrelevant in practice.
However, when vectors are not normalized, the dot product incorporates magnitude information. This can be useful in some settings. For example, in recommendation systems, the magnitude of a user or item embedding might encode popularity or confidence, and the dot product captures both the relevance (direction) and strength (magnitude) of the match.
Euclidean distance (L2 distance) between vectors A and B is:
d(A, B) = sqrt(sum((A_i - B_i)^2))
For L2-normalized vectors, Euclidean distance and cosine similarity are monotonically related:
d(A, B)^2 = 2 - 2 * cos(theta) = 2 * (1 - cos(theta))
This means that for normalized vectors, ranking by cosine similarity and ranking by Euclidean distance produce identical results. The choice between them in this case is purely a matter of convention.
For non-normalized vectors, the relationship breaks down. Euclidean distance is sensitive to magnitude: two vectors pointing in the same direction but with different magnitudes will have a large Euclidean distance despite having cosine similarity of 1.0.
| Property | Cosine Similarity | Dot Product | Euclidean Distance |
|---|---|---|---|
| Range | [-1, 1] | (-inf, +inf) | [0, +inf) |
| Magnitude invariant | Yes | No | No |
| Equivalent for normalized vectors | Yes (= dot product) | Yes (= cosine sim) | Yes (monotonically related) |
| Interpretability | High (bounded, intuitive) | Moderate | Moderate |
| Common use case | Text embeddings, semantic search | Recommendations, attention scores | Clustering, kNN |
| Captures direction only | Yes | No | No |
The standard self-attention mechanism in transformers computes attention scores as the scaled dot product between query and key vectors: score = Q K^T / sqrt(d_k). This is closely related to but distinct from cosine similarity. The scaling by sqrt(d_k) prevents large dot product values but does not normalize by vector magnitudes [8].
Some architectures have experimented with using explicit cosine similarity for attention. In particular, query-key normalization (where queries and keys are L2-normalized before computing attention scores) has been explored to stabilize training and reduce the need for learning rate warmup. This makes the attention logits exactly cosine similarities (up to scaling), bounding them to the range [-1, 1] before the temperature scaling.
Cosine similarity computation consists of three parts: a dot product (O(n) for n-dimensional vectors), and two norm computations (also O(n) each). The total is O(n), making it linear in the vector dimension. For batch computations, cosine similarity matrices between m query vectors and k database vectors can be computed as a matrix multiplication (after normalizing each vector), leveraging highly optimized BLAS routines and GPU tensor operations.
A common optimization is to L2-normalize all vectors once at indexing time. After normalization, cosine similarity reduces to a dot product, eliminating the per-query normalization cost. Most vector databases and embedding libraries apply this optimization by default when cosine similarity is selected as the metric.
When computing cosine similarity in floating-point arithmetic, care must be taken with very similar vectors. The difference between cosine similarity values of 0.9999 and 0.99999 can be significant for ranking but may be lost to floating-point rounding, especially in float16 or bfloat16. For precision-sensitive applications (such as deduplication), using float32 for similarity computation is recommended even when embeddings are stored in lower precision.
For databases with millions or billions of vectors, exact cosine similarity search (comparing the query to every vector) becomes impractical. ANN algorithms trade a small amount of accuracy for orders-of-magnitude speedup:
| ANN Algorithm | Approach | Typical Recall@10 | Build Time |
|---|---|---|---|
| HNSW | Graph-based navigation | 95-99% | Moderate |
| IVF-PQ | Clustering + product quantization | 85-95% | Fast |
| ScaNN | Anisotropic vector quantization | 95-99% | Fast |
| FAISS (flat) | Exact brute force | 100% | N/A (no index) |
These algorithms are designed to find vectors with high cosine similarity to a query without examining every vector in the database. They work by organizing vectors into data structures that allow pruning large portions of the search space.
Despite its widespread use, cosine similarity has several known limitations:
Cosine similarity has been used in information retrieval since the 1960s as part of the Vector Space Model (VSM) proposed by Gerard Salton [10]. In the VSM, documents and queries are represented as term-frequency vectors, and relevance is measured by cosine similarity. This was one of the earliest applications of geometric reasoning to text, and the fundamental idea (represent text as vectors, compare by angle) remains at the core of modern semantic search, just with neural embeddings replacing term-frequency vectors.
The measure gained renewed importance in the deep learning era with the rise of word embeddings, sentence embeddings, and multimodal embeddings, all of which use cosine similarity as their primary comparison metric. Today, cosine similarity underlies billions of daily similarity computations in search engines, recommendation systems, and AI applications worldwide.