Vector embeddings: Difference between revisions

Vector embeddings (view source)

Revision as of 14:35, 8 April 2023

2,092 bytes added , 8 April 2023

no edit summary

Daikon Radish

370

edits

@@ Line 1: / Line 1: @@
 ==Introduction==
-Vector embeddings are numeric representations of data that effectively capture certain features or aspects of the data. In the context of text data, they enable semantic search by representing the meanings of words or phrases. Machine learning models generate these embeddings, which are arrays of real numbers with a fixed length, usually ranging from hundreds to thousands of elements.
+Vector embeddings are numerical representations of data that encapsulate particular features of the data. These embeddings enable the effective execution of semantic search by capturing the semantic similarity between different data objects.
-==Text Data and Vectorization==
+==Understanding Vector Embeddings==
-Vectorization is the process of generating a vector for a data object. For example, two similar words like "cat" and "kitty" may have very different character sequences but share a close meaning. Vectorizing these words might result in highly similar vectors, while vectors for unrelated words like "banjo" or "comedy" would be considerably different.
+In the context of text data, words with similar meanings, such as "cat" and "kitty", must be represented in a manner that captures their semantic similarity. Vector representations achieve this by transforming data objects into arrays of real numbers with a fixed length, typically ranging from hundreds to thousands of elements. These arrays are generated by machine learning models through a process called vectorization.
-Vectors can represent meaning to a certain extent, although it may not be clear what each number in the vector signifies. However, by correlating vectors to familiar words, we can gain a rough understanding of the relationships between them. One notable example is the equation "king − man + woman ≈ queen", which demonstrates that vector embeddings can capture semantic similarities and relationships.
+For instance, the words "cat" and "kitty" may be vectorized as follows:
-Vector embeddings can represent more than just the meanings of words; they can be generated from various types of data, including text, images, audio, time series data, 3D models, videos, and molecules. The distance between two vectors in vector space can be calculated in multiple ways, with one simple method being the sum of the absolute differences between elements at corresponding positions in each vector.
+<code>
+cat = [1.5, -0.4, 7.2, 19.6, 3.1, ..., 20.2]
+kitty = [1.5, -0.4, 7.2, 19.5, 3.2, ..., 20.8]
+</code>
-==Generation of Vector Embeddings==
+These vectors exhibit a high similarity, while vectors for words like "banjo" or "comedy" would not be similar to either of these. In this way, vector embeddings capture the semantic similarity of words. The specific meaning of each number in a vector depends on the machine learning model that generated the vectors, and is not always clear in terms of human understanding of language and meaning.
-The effectiveness of vector embeddings primarily depends on how they are generated for each entity and query. For text data, vectorization techniques have evolved significantly over the last decade, from the introduction of word2vec in 2013 to the state-of-the-art transformer models like BERT, which emerged in 2018.
+Vector-based representation of meaning has gained attention due to its ability to perform mathematical operations between words, revealing semantic relationships. A famous example is:
+<code>
+"king − man + woman ≈ queen"
+</code>
+This result suggests that the difference between "king" and "man" represents some sort of "royalty", which is analogously applicable to "queen" minus "woman". Various concepts, such as "woman", "girl", "boy", etc., can be vectorized into arrays of numbers, often referred to as dimensions. These arrays can be visualized and correlated to familiar words, giving insight into their meaning.
+Vector embeddings can represent more than just word meanings. They can effectively be generated from any data object, including text, images, audio, time series data, 3D models, video, and molecules. Embeddings are constructed such that two objects with similar semantics have vectors that are "close" to each other in vector space, with a "small" distance between them.
+==Generating Vector Embeddings==
+The primary aspect of vector search's effectiveness lies in generating embeddings for each entity and query. The secondary aspect is efficiently searching within very large datasets.
+Vector embeddings can be generated for various media types, such as text, images, audio, and others. For text, vectorization techniques have significantly evolved over the last decade, from word2vec (2013) to the state-of-the-art transformer models era, which began with the release of BERT in 2018.
 ===Word-level Dense Vector Models (word2vec, GloVe, etc.)===
-Word2vec, a family of model architectures, introduced the concept of "dense" vectors in language processing. It uses a neural network model to learn word associations from a large corpus of text, creating a vocabulary and learning vector representations for each word, typically with 300 dimensions. Words found in similar contexts have vector representations that are close in vector space. However, word2vec has some limitations, such as not addressing polysemantic words (words with multiple meanings) or words with ambiguous meanings.
+word2vec is a group of model architectures that introduced the concept of "dense" vectors in language processing, in which all values are non-zero. It uses a neural network model to learn word associations from a large text corpus. The model first creates a vocabulary from the corpus and then learns vector representations for the words, usually with 300 dimensions. Words found in similar contexts have vector representations that are close in vector space.
+However, word2vec suffers from limitations, including its inability to address words with multiple meanings (polysemantic) and words with ambiguous meanings.
 ===Transformer Models (BERT, ELMo, and others)===
-Transformer models like BERT and its successors improve search accuracy, precision, and recall by generating full contextual embeddings for each word, taking the entire input text into account. This allows them to better represent polysemantic words and disambiguate meanings based on context. Some downsides of transformer models include increased compute and memory requirements.
+The current state-of-the-art models are based on the transformer architecture. Models like BERT and its successors improve search accuracy, precision, and recall by examining the context of each word to create full contextual embeddings. Unlike word2vec embeddings, which are context-agnostic, transformer-generated embeddings consider the entire input text. Each occurrence of a word has its own embedding that is influenced by the surrounding text, better reflecting the polysemantic nature of words, which can only be disambiguated when considered in context.
+Some potential downsides of transformer models include:
+Increased compute requirements: Fine-tuning transformer models is much slower (taking hours instead of minutes).
+Increased memory requirements: Context-sensitivity greatly increases memory requirements, often leading to limitations on possible input lengths.
+Despite these drawbacks, transformer models have been incredibly successful, leading to a proliferation of text vectorizer models for various data types such as audio, video, and images. Some models, like CLIP, can vectorize multiple data types (e.g., images and text) into a single vector space, enabling content-based image searches using only text.
+==Vector Embeddings with Weaviate==
+Weaviate is designed to support a wide range of vectorizer models and vectorizer service providers. Users can bring their own vectors, for example, if they already have a vectorization pipeline available or if none of the publicly available models are suitable.
+Weaviate supports using any Hugging Face models through the text2vec-huggingface module, allowing users to choose from many sentence transformers published on Hugging Face. Other popular vectorization APIs, such as OpenAI or Cohere, can be used through the text2vec-openai or text2vec-cohere modules. Users can also run transformer models locally with text2vec-transformers, and modules like multi2vec-clip can convert images and text to vectors using a CLIP model.
-==Vector Embeddings in Weaviate==
+All of these models perform the same core task, which is to represent the "meaning" of the original data as a set of numbers, enabling the effective implementation of semantic search.
-Weaviate is designed to support various vectorizer models and service providers, allowing users to bring their own vectors or use publicly available models. It supports Hugging Face models through the text2vec-huggingface module, enabling the use of many sentence transformers available on the platform. Other popular vectorization APIs, such as OpenAI and Cohere, are supported through the text2vec-openai and text2vec-cohere modules, respectively. Users can also run transformer models locally with text2vec-transformers or use multi2vec-clip to convert images and text to vectors using a CLIP model.