# HuggingGPT

> Source: https://aiwiki.ai/wiki/hugginggpt
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: AI Agents, AI Research, Multimodal AI
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "HuggingGPT." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/hugginggpt
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

HuggingGPT is an agent system that uses a [large language model](https://aiwiki.ai/wiki/large_language_model) as a controller to plan tasks and orchestrate specialist machine learning models hosted on [Hugging Face](https://aiwiki.ai/wiki/hugging_face). It was introduced in the March 2023 paper "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face" by Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang, a collaboration between [Zhejiang University](https://aiwiki.ai/wiki/zhejiang_university) and [Microsoft Research](https://aiwiki.ai/wiki/microsoft_research) Asia [1]. The paper was accepted to the main conference track of [NeurIPS](https://aiwiki.ai/wiki/neurips) 2023 [2].

The core idea is to treat language as a generic interface between an LLM and the rest of the machine learning ecosystem. When a user submits a request, [ChatGPT](https://aiwiki.ai/wiki/chatgpt) decomposes it into subtasks, picks a suitable Hugging Face model for each subtask based on the models' published descriptions, runs the models, and then writes a natural-language answer that integrates their outputs [1]. Because the expert models cover vision, speech, and video as well as text, the combined system can answer [multimodal](https://aiwiki.ai/wiki/multimodal_model) requests that no single chat model of the time could handle alone, such as counting the animals in an image, generating a new picture of them, and reading the description aloud.

Microsoft published the implementation on GitHub in a repository named JARVIS, and the system is often referred to by that name [3]. Alongside contemporaneous work such as [Toolformer](https://aiwiki.ai/wiki/toolformer), Visual ChatGPT, and AutoGPT, HuggingGPT became one of the reference points for the "LLM as controller" design that later hardened into mainstream [tool use](https://aiwiki.ai/wiki/tool_use) and agent frameworks [5].

## Background

In early 2023, ChatGPT and [GPT-4](https://aiwiki.ai/wiki/gpt-4) were demonstrating strong language ability, but they could not natively see images, hear audio, or perform specialized prediction tasks. One response was to train bigger multimodal models. Another was to let the LLM call external tools and models, a line of work that moved quickly that spring: Toolformer (February 9, 2023) showed a language model teaching itself to call APIs such as calculators and search engines [6], Microsoft's Visual ChatGPT (March 8, 2023) wired ChatGPT to a fixed set of visual foundation models [7], and TaskMatrix.AI (March 29, 2023) sketched a platform connecting foundation models to millions of APIs [8].

HuggingGPT, first posted to arXiv on March 30, 2023, pushed the same idea in a different direction: instead of a hand-picked toolbox, the controller draws on the open Hugging Face Hub, where thousands of community models are already organized by task and described by model cards [1]. The work was done while first author Yongliang Shen was an intern at Microsoft Research Asia [1]. The paper went through several arXiv revisions during 2023, with the fourth version posted on December 3, 2023, around the time of its NeurIPS publication [1][2].

## How it works

HuggingGPT runs a four-stage pipeline for every request. The LLM (the paper's experiments use gpt-3.5-turbo, text-davinci-003, and GPT-4) does the reasoning at stages 1, 2, and 4, while stage 3 is conventional model inference [1].

| Stage | What happens |
|---|---|
| Task planning | The LLM parses the user request into a list of structured subtasks with dependencies between them |
| Model selection | For each subtask, the LLM picks an expert model from candidates on the Hugging Face Hub, based on their descriptions |
| Task execution | The selected models run on the subtask inputs and return results |
| Response generation | The LLM integrates all intermediate results into a natural-language answer for the user |

Task planning relies on two prompt-engineering techniques. Specification-based instruction forces the plan into a JSON template in which every subtask has four slots: "task" (the task type), "id" (a unique identifier), "dep" (the ids of subtasks it depends on), and "args" (its arguments). Demonstration-based parsing adds worked examples to the prompt, a form of [in-context learning](https://aiwiki.ai/wiki/in_context_learning) that helps the model produce well-formed plans and capture the logical connections between subtasks [1].

Model selection cannot simply paste the whole Hub into the prompt. HuggingGPT first filters candidate models by the subtask's task type, ranks them by download count on Hugging Face, and keeps only the top-K so that their ids and descriptions fit in the context window. The LLM then chooses among these candidates in context, which the authors describe as dynamic task-model assignment: new models become usable as soon as their descriptions appear in the candidate list, without any retraining [1].

Task execution handles the plumbing between subtasks. Where one subtask consumes another's output, the plan refers to the future result with a placeholder written as "<resource>-task_id", which the executor replaces with the real output once the prerequisite finishes; subtasks without dependencies between them can run in parallel. Execution supports hybrid inference endpoints: models deployed locally are preferred for speed, with Hugging Face's hosted inference endpoints as the fallback [1]. Finally, response generation feeds the plan and all intermediate results back to the LLM, which writes the answer and can explain which models produced which piece of it [1].

The system supports 24 task types across language, vision, speech, and video, including image captioning, [object detection](https://aiwiki.ai/wiki/object_detection), [visual question answering](https://aiwiki.ai/wiki/visual_question_answering_models), text-to-image generation, pose detection, image segmentation, and [text-to-speech](https://aiwiki.ai/wiki/text_to_speech_ai) [1].

## Evaluation

Evaluating open-ended orchestration was itself a research problem, and the authors leaned on GPT-4 as an automatic judge, an early instance of the [LLM-as-a-judge](https://aiwiki.ai/wiki/llm_as_a_judge) pattern. They built a dataset of 3,497 user requests with GPT-4-generated plans as pseudo labels, split into three structural classes (1,450 single-task, 1,917 sequential-task, and 130 graph-task requests), plus a smaller set of 46 complex examples with human-expert plans as ground truth; against the human annotations the LLM planners still showed a substantial gap [1].

On the GPT-4-annotated benchmark, planning quality tracked the strength of the controller model. The paper reports the following task-planning results for gpt-3.5-turbo against two 7B open-source baselines [1]:

| Controller | Single task (accuracy) | Sequential tasks (F1) | Graph tasks (F1) |
|---|---|---|---|
| GPT-3.5 | 52.62% | 51.92% | 51.91% |
| [Vicuna](https://aiwiki.ai/wiki/vicuna)-7b | 23.86% | 22.89% | 18.66% |
| [Alpaca](https://aiwiki.ai/wiki/alpaca)-7b | 6.48% | 22.80% | 20.59% |

In a separate human evaluation on 130 diverse requests, invited experts judged [GPT-3.5](https://aiwiki.ai/wiki/gpt-3.5)-based HuggingGPT to produce a passing task plan 91.22% of the time, while 63.08% of runs ended in an overall successful response, illustrating how errors compound across the pipeline stages [1].

## The JARVIS repository

The open-source release lives at github.com/microsoft/JARVIS under an MIT license and had accumulated roughly 25,000 GitHub stars by mid-2026 [3]. In April 2023 the project added Azure [OpenAI](https://aiwiki.ai/wiki/openai) support, GPT-4 compatibility, and a Gradio demo that was also hosted as a Hugging Face Space; a rebuild and [LangChain](https://aiwiki.ai/wiki/langchain) integration followed in July 2023 [3]. The code can run in several configurations, from a lite mode that calls only hosted Hugging Face inference endpoints and downloads no local models, through hybrid setups, to a full local deployment whose default configuration calls for a GPU with at least 24 GB of VRAM, up to 80 GB of RAM, and more than 284 GB of disk space for the expert models. A command-line interface and a REST-style server are included [3]. The hosted demo has not been maintained; as of July 2026 the microsoft/HuggingGPT Space shows a runtime error rather than a working interface [4].

The repository later grew into an umbrella for the group's task-automation research. TaskBench (released November 30, 2023, and published at NeurIPS 2024) is a benchmark from an overlapping author team that formalizes exactly the abilities HuggingGPT exercises, scoring LLMs on task decomposition, tool selection, and parameter prediction over "tool graphs" [3][9]. EasyTool, added in January 2024, distills long and inconsistent tool documentation into concise instructions to make tool-using agents more reliable [3].

## Significance and relation to later agent work

HuggingGPT is one of the earliest and most cited demonstrations that a chat LLM can serve as the [planning](https://aiwiki.ai/wiki/agent_planning) and coordination layer for other models rather than as a standalone answer engine; by July 2026 Semantic Scholar counted about 1,600 citations of the paper [10]. Lilian Weng's widely read June 2023 survey of LLM-powered [autonomous agents](https://aiwiki.ai/wiki/ai_agents) used it as the worked example of the tool-use component, alongside planning and memory [5].

Pieces of its design became standard practice. Forcing plans and tool calls into a JSON schema anticipated the structured [function calling](https://aiwiki.ai/wiki/function_calling) interfaces that model providers later built into their APIs, and selecting tools at run time from a described catalog, rather than baking a fixed toolbox into the prompt, is the same pattern that later tool ecosystems built around registries of tool descriptions. Where fully autonomous agents such as [AutoGPT](https://aiwiki.ai/wiki/autogpt) loop an LLM against open-ended goals, HuggingGPT kept a bounded, interpretable pipeline: one plan, one selection step, one execution pass, one answer. That single-pass structure limits what it can recover from mid-run, but it also made the system easy to study, and the follow-up TaskBench benchmark turned its stage decomposition into a measurement instrument for other models [9].

## Limitations

The paper is candid about the system's weak points [1]:

- Efficiency and latency. Every request needs multiple LLM inferences (planning, selection, response generation) on top of the expert-model calls, so responses are slow compared with a single model answering directly.
- Context length. The number of candidate models, the length of their descriptions, and the number of demonstrations must all fit in the controller's context window, which caps how much of the Hub the system can actually consider.
- Planning quality. The whole pipeline trusts the LLM's plan, and plans can be wrong or suboptimal; the task-planning scores above show even GPT-3.5 getting barely half of them right on the harder splits.
- Stability. LLMs sometimes fail to follow the output format or otherwise misbehave, and any exception in a middle stage breaks the run.

A further practical dependence is on Hub metadata: model selection ranks candidates by download counts and reasons over model card descriptions, so poorly documented or unpopular models are effectively invisible to the controller regardless of their quality [1].

## See also

- [Tool use](https://aiwiki.ai/wiki/tool_use)
- [Toolformer](https://aiwiki.ai/wiki/toolformer)
- [AutoGPT](https://aiwiki.ai/wiki/autogpt)
- [LangChain](https://aiwiki.ai/wiki/langchain)
- [AI agents](https://aiwiki.ai/wiki/ai_agents)
- [Hugging Face](https://aiwiki.ai/wiki/hugging_face)

## References

1. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y. "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face." arXiv:2303.17580, first posted March 30, 2023, v4 December 3, 2023. https://arxiv.org/abs/2303.17580
2. NeurIPS Proceedings. "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face." Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Main Conference Track. https://proceedings.neurips.cc/paper_files/paper/2023/hash/77c33e6a367922d003ff102ffb92b658-Abstract-Conference.html
3. Microsoft. "JARVIS" GitHub repository (HuggingGPT, TaskBench, EasyTool). https://github.com/microsoft/JARVIS
4. Hugging Face Spaces. "microsoft/HuggingGPT" (demo Space, runtime error as of July 2026). https://huggingface.co/spaces/microsoft/HuggingGPT
5. Weng, L. "LLM Powered Autonomous Agents." Lil'Log, June 23, 2023. https://lilianweng.github.io/posts/2023-06-23-agent/
6. Schick, T., et al. "Toolformer: Language Models Can Teach Themselves to Use Tools." arXiv:2302.04761, February 9, 2023. https://arxiv.org/abs/2302.04761
7. Wu, C., et al. "Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models." arXiv:2303.04671, March 8, 2023. https://arxiv.org/abs/2303.04671
8. Liang, Y., et al. "TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs." arXiv:2303.16434, March 29, 2023. https://arxiv.org/abs/2303.16434
9. Shen, Y., Song, K., Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., Zhuang, Y. "TaskBench: Benchmarking Large Language Models for Task Automation." arXiv:2311.18760, November 30, 2023; NeurIPS 2024. https://arxiv.org/abs/2311.18760
10. Semantic Scholar. Paper record for arXiv:2303.17580 (citation count retrieved July 2026). https://api.semanticscholar.org/graph/v1/paper/arXiv:2303.17580?fields=title,venue,year,citationCount