ChatGPT Plugins

See also: ChatGPT

Introduction

Although language models like ChatGPT have proven valuable in various applications, they continue to be constrained by their dependence on training data. To augment their abilities, plugins can act as "sensors" for language models, supplying current, individual, or specific information not found in their training data. Additionally, plugins allow language models to carry out secure and controlled actions based on user requests, enhancing the system's overall utility.

Browsing

Influenced by prior projects like WebGPT, GopherCite, BlenderBot2, LaMDA2, and more, browsing-capable language models can retrieve information from the web, broadening their knowledge beyond their training dataset. This enables ChatGPT to engage with current information and deliver more pertinent responses to user inquiries.

Code Interpreter

The code interpreter supplies ChatGPT with a functional Python interpreter within a protected, firewalled execution environment, as well as temporary disk space. The interpreter plugin assesses code in a persistent session that remains active throughout a chat conversation. Users can upload files to the workspace and download their work results.

The incorporation of a code interpreter intends to establish a natural interface for computer functionalities, streamlining workflows and expanding accessibility. Preliminary user studies have recognized several use cases for the code interpreter:

• Resolving mathematical problems
• Conducting data analysis and visualization
• Converting files between different formats

Retrieval

The open-source retrieval plugin (GitHub Link) allows ChatGPT access private or organizational information sources, including files, notes, emails, or public documentation, upon user authorization. Users can acquire pertinent document excerpts from their data sources by posing questions or conveying requirements in natural language.

Developers can deploy their own version of the plugin, which utilizes OpenAI embeddings and collaborates with an array of vector databases (Milvus, Pinecone, Qdrant, Redis, Weaviate, or Zilliz) for document indexing and searching. Information sources can be synchronized with the database via webhooks.

Developers can begin by visiting the retrieval plugin repository.

Third-Party Plugins

Third-party plugins are characterized by a manifest file that contains a machine-readable outline of the plugin's features, execution guidelines, and user-oriented documentation. The process of developing a plugin comprises the following stages:

1. Design an API with endpoints that the language model can access. This can be a new API, a pre-existing API, or a wrapper around an existing API specifically crafted for language models.
2. Generate an OpenAPI specification to outline the API and a manifest file that connects to the OpenAPI spec and comprises plugin-related metadata.

When initiating a conversation on chat.openai.com, users can choose which third-party plugins to activate. The language model is presented with documentation about the enabled plugins as part of the conversation context, permitting the model to call suitable plugin APIs as required to satisfy user intentions. Although current plugins are designed for accessing backend APIs, the investigation of plugins able to call client-side APIs continues.

List of Third-Party Plugins

Plugin	Description
Expedia	Bring your trip plans to life—get there, stay there, find things to see and do.
FiscalNote	Provides and enables access to select market-leading, real-time data sets for legal, political, and regulatory data and information.
Instacart	Order from your favorite local grocery stores.
KAYAK	Search for flights, stays and rental cars. Get recommendations for all the places you can go within your budget.
Klarna Shopping	Search and compare prices from thousands of online shops.
Milo Family AI	Giving parents superpowers to turn the manic to magic, 20 minutes each day. Ask: Hey Milo, what’s magic today?
OpenTable	Provides restaurant recommendations, with a direct link to book.
Shop	Search for millions of products from the world’s greatest brands.
Speak	Learn how to say anything in another language with Speak, your AI-powered language tutor.
Wolfram	Alpha and Wolfram Language.
Zapier	Interact with over 5,000+ apps like Google Sheets, Trello, Gmail, HubSpot, Salesforce, and more.

Documentation

OpenAI ChatGPT Plugins Documentation