370
edits
Vector database: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
A [[vector database]] is a type of [[database]] specifically designed for storing and querying [[high-dimensional vector data]], which is often used in [[artificial intelligence applications]] ([[AI]] [[apps]]). Complex data, including unstructured forms like documents, images, videos, and plain text, is growing rapidly. Traditional databases designed for structured data struggle to effectively store and analyze complex data, often requiring extensive keyword and metadata classification. Vector databases address this issue by transforming complex data into vector embeddings, which describe data objects in numerous dimensions. These databases are gaining popularity due to their ability to extend [[large language models]] ([[LLMs]]) with [[long-term memory]] and provide efficient [[querying]] for AI-driven applications. | |||
== | ==Vector Embeddings== | ||
[[Vector]]s are arrays of numbers that can represent complex objects such as [[words]], [[sentences]], [[images]], or [[audio]] files in a [[continuous high-dimensional space]], called an [[embedding]]. [[Embeddings]] work by mapping [[semantically similar]] words or features from various data types together. These embeddings can be used in [[recommendation systems]], [[search engines]], and [[text generation applications]] like [[ChatGPT]]. | |||
==Database Structure== | |||
In a [[relational database]], data is organized in rows and columns, while in a [[document database]], it is organized in documents and collections. In contrast, a vector database stores arrays of numbers that are clustered based on [[similarity]]. These databases can be queried with [[ultra-low latency]], making them ideal for AI-driven applications. | |||
==Vector Database Products== | |||
Several vector databases have emerged to cater to the growing demand for AI applications. Some of the popular native vector databases include open-source options like [[Weaviate]] and [[Milvus]], both written in Go. [[Pinecone]] is another popular vector database, although it is not open source. [[Chroma]], based on Clickhouse, is an open-source project with a growing following. Relational databases like [[Postgres]] have tools like [[pgVector]], and [[Redis]] has first-class vector support to accommodate this type of functionality. | |||
==Using Vector Databases with Large Language Models== | |||
One of the primary reasons for the increasing popularity of vector databases is their ability to extend large language models (LLMs) with long-term memory. By providing a general-purpose model, such as [[OpenAI]]'s [[GPT-4]], [[Meta]]'s [[LLaMA]], or [[Google]]'s [[LaMDA]], users can store their own data in a vector database. When [[prompt]]ed, the database can query relevant documents to update the context, customizing the final response and providing the AI with long-term memory. | |||
In addition, vector databases can integrate with tools like [[LangChain]], which combine multiple LLMs together for more advanced applications. | |||
==Why Use a Vector Database?== | ==Why Use a Vector Database?== | ||
Vector databases offer several use cases, including: | Vector databases offer several use cases, including: | ||
| Line 20: | Line 35: | ||
===Deduplication and record matching=== | ===Deduplication and record matching=== | ||
Vector similarity search can be used to find near-duplicate records for applications such as removing duplicate items from a catalog | Vector similarity search can be used to find near-duplicate records for applications such as removing duplicate items from a catalog | ||
===Anomaly detection=== | ===Anomaly detection=== | ||