370
edits
No edit summary |
No edit summary |
||
Line 2: | Line 2: | ||
{{see also|AI terms}} | {{see also|AI terms}} | ||
==Introduction== | ==Introduction== | ||
A [[vector database]] is a type of [[database]] specifically designed for storing and querying [[high-dimensional vector data]], which is often used in [[artificial intelligence applications]] ([[AI]] [[apps]]). Complex data, including unstructured forms like documents, images, videos, and plain text, is growing rapidly. Traditional databases designed for structured data struggle to | A [[vector database]] is a type of [[database]] specifically designed for storing and querying [[high-dimensional vector data]], which is often used in [[artificial intelligence applications]] ([[AI]] [[apps]]). Complex data, including unstructured forms like documents, images, videos, and plain text, is growing rapidly. Traditional databases designed for structured data struggle to store and analyze complex data effectively, often requiring extensive keyword and metadata classification. Vector databases address this issue by transforming complex data into [[vector embeddings]], which describe data objects in numerous dimensions. These databases are gaining popularity due to their ability to extend [[large language models]] ([[LLMs]]) with [[long-term memory]] and provide efficient [[querying]] for [[artificial intelligence applications|AI-driven applications]]. | ||
==What is a Vector Database?== | ==What is a Vector Database?== | ||
In a [[relational database]], data is organized in rows and columns, while in a [[document database]], it is organized in documents and collections. In contrast, a vector database stores arrays of numbers | In a [[relational database]], data is organized in rows and columns, while in a [[document database]], it is organized in documents and collections. In contrast, a vector database stores arrays of numbers clustered based on [[similarity]]. These databases can be queried with [[ultra-low latency]], making them ideal for AI-driven applications. | ||
A vector database is a type of database that indexes and stores vector | A vector database is a type of database that indexes and stores [[vector embedding]]s for efficient [[retrieval]] and [[similarity search]]. In addition to traditional [[CRUD]] (create, read, update, and delete) operations and metadata filtering, vector databases enable the organization and comparison of any vector to one another or to the vector of a search query. This capability allows vector databases to excel at similarity search or [[vector search]], providing more comprehensive search results that would not be possible with traditional search technology. | ||
==Vector Database Products== | ==Vector Database Products== | ||
Several vector databases have emerged to cater to the growing demand for AI applications. Some of the popular native vector databases include open-source options like [[Weaviate]] and [[Milvus]], both written in Go. [[Pinecone]] is another popular vector database, although it is not open source. [[Chroma]], based on Clickhouse, is an open-source project with a growing following. Relational databases like [[Postgres]] have tools like [[pgVector]], and [[Redis]] has first-class vector support to accommodate this type of functionality. | Several vector databases have emerged to cater to the growing demand for AI applications. Some of the popular native vector databases include open-source options like '''[[Weaviate]]''' and '''[[Milvus]]''', both written in Go. '''[[Pinecone]]''' is another popular vector database, although it is not open source. '''[[Chroma]]''', based on Clickhouse, is an open-source project with a growing following. Relational databases like [[Postgres]] have tools like [[pgVector]], and [[Redis]] has first-class vector support to accommodate this type of functionality. | ||
==Using Vector Databases with Large Language Models== | ==Using Vector Databases with Large Language Models== | ||
One of the primary reasons for the increasing popularity of vector databases is their ability to extend large language models (LLMs) with long-term memory. By providing a general-purpose model, such as [[OpenAI]]'s [[GPT-4]], [[Meta]]'s [[LLaMA]], or [[Google]]'s [[LaMDA]], users can store their own data in a vector database. When [[prompt]]ed, the database can query relevant documents to update the context, customizing the final response and providing the AI with long-term memory. | One of the primary reasons for the increasing popularity of vector databases is their ability to extend large language models (LLMs) with [[long-term memory]]. By providing a general-purpose model, such as [[OpenAI]]'s [[GPT-4]], [[Meta]]'s [[LLaMA]], or [[Google]]'s [[LaMDA]], users can store their own data in a vector database. When [[prompt]]ed, the database can query relevant documents to update the context, customizing the final response and providing the AI with long-term memory. | ||
In addition, vector databases can integrate with tools like [[LangChain]], which combine multiple LLMs together for more advanced applications. | In addition, vector databases can integrate with tools like [[LangChain]], which combine multiple LLMs together for more advanced applications. | ||
Line 19: | Line 19: | ||
==Why Use a Vector Database?== | ==Why Use a Vector Database?== | ||
===Semantic search=== | ===Semantic search=== | ||
Unlike lexical search, which relies on exact word or string matches, semantic search uses the meaning and context of a search query or question. Vector databases use Natural Language Processing models to store and index vector embeddings, allowing for more accurate and relevant search results. | Unlike [[lexical search]], which relies on exact word or string matches, [[semantic search]] uses the meaning and context of a search query or question. Vector databases use [[Natural Language Processing]] [[models]] to store and index vector embeddings, allowing for more accurate and relevant search results. | ||
===Similarity search for unstructured data=== | ===Similarity search for unstructured data=== | ||
Vector databases facilitate search and retrieval of unstructured data like images, audio, video, and JSON, which can be challenging to classify and store in traditional databases. | Vector databases facilitate the search and retrieval of unstructured data like [[images]], [[audio]], [[video]], and [[JSON]], which can be challenging to classify and store in traditional databases. | ||
===Ranking and recommendation engines=== | ===Ranking and recommendation engines=== | ||
By finding similar items based on nearest matches, vector databases are suitable for powering ranking and recommendation engines for online retailers and streaming media services. | By finding similar items based on nearest matches, vector databases are suitable for powering [[ranking engines|ranking]] and [[recommendation engines]] for online retailers and streaming media services. | ||
===Deduplication and record matching=== | ===Deduplication and record matching=== | ||
Vector similarity search can be used to find near-duplicate records for applications such as removing duplicate items from a catalog | Vector similarity search can be used to find near-duplicate records for applications such as removing duplicate items from a catalog. | ||
===Anomaly detection=== | ===Anomaly detection=== | ||
Vector databases can identify anomalies in applications used for threat assessment, fraud detection, and IT operations by finding objects that are distant or dissimilar from expected results. | Vector databases can identify [[anomalies]] in applications used for threat assessment, fraud detection, and IT operations by finding objects that are distant or dissimilar from expected results. | ||
== | ==Features a Vector Database== | ||
===Vector Indexes for Search and Retrieval=== | |||
Vector databases employ algorithms to [[index vectors|index]] and [[retrieve vectors]] efficiently. Accuracy, latency, or memory usage may need to be prioritized depending on specific use cases. [[Common similarity]] and [[distance metrics]] used in [[vector index]]es are [[Euclidean distance]], [[cosine similarity]], and [[dot product]]s. | |||
[[Approximate Nearest Neighbor]] ([[ANN]]) search is a popular technique to balance precision and performance. ANN algorithms, such as [[HNSW]], [[IVF]], or [[PQ]], focus on improving specific performance properties like memory reduction or fast and accurate search times. [[Composite index]]es combine several components and are often used to achieve optimal performance for a given use case. | |||
Building an effective index without a vector database can be challenging and may require a team of experienced engineers with expertise in [[index algorithm|indexing]] and [[retrieval algorithm]]s. | |||
===Single-Stage Filtering=== | ===Single-Stage Filtering=== | ||
Single-stage filtering combines the accuracy of pre-filtering with the speed of post-filtering | [[Single-stage filtering]] is essential for effective vector databases, as it enables users to limit search results based on [[vector metadata]]. It combines the accuracy of [[pre-filtering]] with the speed of [[post-filtering]], merging [[vector index|vector]] and [[metadata index]]es into a single index for optimal performance. | ||
=== | ===Data Sharding === | ||
Scaling is critical for vector databases to handle large volumes of data. [[Data sharding]] allows the database to divide vectors into shards and replicas across multiple machines, providing scalable and cost-effective performance. When searching, the database queries each shard and combines the results to determine the best match. This can be achieved using Kubernetes, with each shard assigned its own pod containing CPU and RAM resources. | |||
=== | ===Replication=== | ||
[[Replication]] is necessary for vector databases to handle multiple requests simultaneously or in rapid succession. By replicating the set of [[pods]], more requests can be processed in parallel. Replicas also improve availability, as they can be spread across different availability zones provided by cloud providers, ensuring high availability even when machines fail. | |||
== | ===Hybrid Storage=== | ||
[[Hybrid storage]] configurations store a compressed vector index in memory (RAM) and the original, full-resolution vector index on disk. This approach reduces infrastructure costs while maintaining fast and accurate search results. Hybrid storage increases storage capacity without negatively impacting database performance. | |||
=== | ===API=== | ||
[[APIs]] enable developers to use and manage vector databases from other applications, offloading the burden of building and maintaining vector search capabilities. [[REST API]]s allow vector databases to be accessed from any environment capable of making HTTPS calls, while direct access can be provided through clients using languages like [[Python]], [[Java]], and [[Go]]. | |||
[[Category:Terms]] [[Category:Artificial intelligence terms]] | [[Category:Terms]] [[Category:Artificial intelligence terms]] |
edits