Jump to content

Vector database: Difference between revisions

no edit summary
No edit summary
No edit summary
Line 1: Line 1:
==Introduction==
==Introduction==
Complex data, which includes unstructured forms like documents, images, videos, and plain text, is growing rapidly. Traditional databases designed for structured data struggle to effectively store and analyze complex data, often requiring extensive keyword and metadata classification. Machine Learning (ML) techniques can address this issue by transforming complex data into vector embeddings, which describe data objects in numerous dimensions.


Vector databases are designed specifically to handle vector embeddings. These databases index vectors, allowing for easy search and retrieval by comparing values and identifying those most similar to one another. Although challenging to implement, there are various solutions available for vector databases, ranging from plugins and open-source projects to fully-managed services.
A [[vector database]] is a type of [[database]] specifically designed for storing and querying [[high-dimensional vector data]], which is often used in [[artificial intelligence applications]] ([[AI]] [[apps]]). Complex data, including unstructured forms like documents, images, videos, and plain text, is growing rapidly. Traditional databases designed for structured data struggle to effectively store and analyze complex data, often requiring extensive keyword and metadata classification. Vector databases address this issue by transforming complex data into vector embeddings, which describe data objects in numerous dimensions. These databases are gaining popularity due to their ability to extend [[large language models]] ([[LLMs]]) with [[long-term memory]] and provide efficient [[querying]] for AI-driven applications.


==What is a Vector Database?==
==Vector Embeddings==
A vector database is a type of database that indexes and stores vector embeddings for efficient retrieval and similarity search. In addition to traditional CRUD (create, read, update, and delete) operations and metadata filtering, vector databases enable the organization and comparison of any vector to one another or to the vector of a search query. This capability allows vector databases to excel at similarity search or "vector search," providing more comprehensive search results that would not be possible with traditional search technology.
 
[[Vector]]s are arrays of numbers that can represent complex objects such as [[words]], [[sentences]], [[images]], or [[audio]] files in a [[continuous high-dimensional space]], called an [[embedding]]. [[Embeddings]] work by mapping [[semantically similar]] words or features from various data types together. These embeddings can be used in [[recommendation systems]], [[search engines]], and [[text generation applications]] like [[ChatGPT]].
 
==Database Structure==
 
In a [[relational database]], data is organized in rows and columns, while in a [[document database]], it is organized in documents and collections. In contrast, a vector database stores arrays of numbers that are clustered based on [[similarity]]. These databases can be queried with [[ultra-low latency]], making them ideal for AI-driven applications.
 
==Vector Database Products==
 
Several vector databases have emerged to cater to the growing demand for AI applications. Some of the popular native vector databases include open-source options like [[Weaviate]] and [[Milvus]], both written in Go. [[Pinecone]] is another popular vector database, although it is not open source. [[Chroma]], based on Clickhouse, is an open-source project with a growing following. Relational databases like [[Postgres]] have tools like [[pgVector]], and [[Redis]] has first-class vector support to accommodate this type of functionality.
 
==Using Vector Databases with Large Language Models==
 
One of the primary reasons for the increasing popularity of vector databases is their ability to extend large language models (LLMs) with long-term memory. By providing a general-purpose model, such as [[OpenAI]]'s [[GPT-4]], [[Meta]]'s [[LLaMA]], or [[Google]]'s [[LaMDA]], users can store their own data in a vector database. When [[prompt]]ed, the database can query relevant documents to update the context, customizing the final response and providing the AI with long-term memory.
 
In addition, vector databases can integrate with tools like [[LangChain]], which combine multiple LLMs together for more advanced applications.


==Why Use a Vector Database?==
==Why Use a Vector Database?==
Vector databases offer several use cases, including:
Vector databases offer several use cases, including:


Line 20: Line 35:


===Deduplication and record matching===
===Deduplication and record matching===
Vector similarity search can be used to find near-duplicate records for applications such as removing duplicate items from a catalog.
Vector similarity search can be used to find near-duplicate records for applications such as removing duplicate items from a catalog


===Anomaly detection===
===Anomaly detection===
370

edits