Vicuna (language model)

Vicuna is a family of chat-tuned, open-weight large language models released by LMSYS in 2023. The first public model, Vicuna-13B v0, was announced on March 30, 2023. It was produced by supervised fine-tuning of Meta's LLaMA on about 70,000 user-shared conversations collected from ShareGPT. LMSYS also released training, serving, and evaluation code through FastChat.^[1]

The initial release attracted attention for a preliminary claim that a GPT-4 judge rated Vicuna-13B at more than 90 percent of the response quality of ChatGPT and Google Bard. LMSYS explicitly described that comparison as "fun and non-scientific" and said that more rigorous evaluation was needed.^[1] The team subsequently developed MT-Bench and Chatbot Arena, two evaluation projects that grew beyond Vicuna itself.^[2] Later Vicuna releases expanded the training set, added a 33-billion-parameter model, and moved the 7B and 13B variants to LLaMA 2.^[3]

Quick facts

Scroll sideways for more →

Origin

Vicuna appeared during the first wave of chat models derived from publicly described foundation models. Meta's LLaMA paper introduced base models from 7B to 65B parameters in February 2023.^[4] Stanford's Alpaca project then fine-tuned LLaMA-7B on 52,000 instruction-response examples generated with a modified Self-Instruct process. Alpaca released its data-generation and training code for research use.^[5]

LMSYS adapted that recipe for multi-turn dialogue. Its team collected conversations that ShareGPT users had chosen to share, converted HTML to Markdown, filtered some inappropriate or low-quality samples, and split long conversations into segments that fit the training context. The initial release used about 70,000 conversations. The project did not release the underlying ShareGPT dataset.^[1]

The named team included Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. LMSYS described the work as a collaboration across five universities.^[1]

Training method

Vicuna used supervised instruction tuning rather than reinforcement learning from human feedback. For a multi-turn conversation, the training code masked the user's text and calculated the token-level cross-entropy loss on the assistant's responses. LMSYS increased the maximum sequence length from Alpaca's 512 tokens to 2,048 and used gradient checkpointing and FlashAttention to reduce memory pressure.^[1]

The initial models were trained with PyTorch Fully Sharded Data Parallel on eight NVIDIA A100 GPUs in about one day. LMSYS reported spot-compute costs of about $140 for the 7B model and $300 for the 13B model. The team used SkyPilot for managed spot instances, recovery from preemption, and switching availability zones.^[1]

The FastChat repository publishes the main 13B training hyperparameters: a global batch size of 128, learning rate of 2e-5, three epochs, maximum length of 2,048, and zero weight decay. Its example command also uses a cosine learning-rate schedule, a 0.03 warmup ratio, bfloat16 training, gradient checkpointing, and FSDP.^[6] These settings document the released recipe, but they do not establish that every Vicuna checkpoint was trained with an identical configuration.

The 2023 MT-Bench paper describes a cleaned set of about 125,000 ShareGPT conversations. It reports three experimental subsets: a 3,000-conversation selected set, a first-turn-only set, and the full set. The full set contained about 370 million training tokens. The paper found that a small selected set quickly improved GPT-4 preference scores, while improvements on MMLU were more closely associated with using more training data.^[2]

Versions

LMSYS's version notes list four public release lines.^[3]

Scroll sideways for more →

The version notes do not list public v1.2 or v1.4 releases, and they do not state why those numbers were skipped. The v1.5 model cards describe an autoregressive transformer fine-tuned from Llama 2 on about 125,000 conversations. The 16K cards say that conversations were packed into 16,000-token sequences and that linear RoPE scaling was used.^[7]

The current FastChat documentation continues to point to v1.5 repositories for the 7B and 13B models, including their 16K variants, and to v1.3 for the 33B model.^[6] This supports identifying v1.5 as the latest documented LMSYS release line, without implying that the weights are competitive with newer model families.

The "90 percent" evaluation

For the original evaluation, LMSYS created 80 questions across eight categories and collected answers from LLaMA-13B, Alpaca-13B, Vicuna-13B, Bard, and ChatGPT. GPT-4 was prompted to assess helpfulness, relevance, accuracy, and detail. The blog reported both preference results and 1-to-10 scores, while noting that GPT-4 was not good at judging coding and mathematics answers.^[1]

The source reports four separate baseline-versus-Vicuna comparisons:

Scroll sideways for more →

The 92 percent headline comes from 638 / 693 in the ChatGPT comparison. The four Vicuna totals are not interchangeable: each came from a different comparison prompt, so the table cannot be converted into a single five-model score ranking. LMSYS also reported that GPT-4 rated Vicuna better than or equal to ChatGPT on 45 percent of the 80 questions.^[1]

Later research demonstrated why the caveat mattered. Wang et al. tested order effects on the same evaluation setting and found that changing response order could greatly alter an LLM referee's result. Their paper reported a manipulated condition in which Vicuna-13B appeared to beat ChatGPT on 66 of the 80 questions, and proposed balanced-position and human-in-the-loop calibration.^[8] This was an evaluation-method finding, not evidence that Vicuna actually surpassed ChatGPT.

A separate study of model imitation by Gudibande et al. found that fine-tuning smaller models on proprietary-model outputs could transfer instruction-following style without closing the same gap on targeted factuality and capability benchmarks. That study addressed the broader imitation approach rather than serving as a direct re-evaluation of every Vicuna checkpoint, but its findings caution against treating fluent style or preference ratings as complete measures of model capability.^[9]

MT-Bench and Chatbot Arena

LMSYS formalized its evaluation work in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." MT-Bench contains 80 two-turn questions divided among writing, roleplay, extraction, reasoning, mathematics, coding, STEM knowledge, and humanities or social-science knowledge. An LLM-as-a-judge system can compare two answers or assign individual scores. The study also documented position bias, verbosity bias, possible self-enhancement bias, and failures on mathematics and reasoning judgments.^[2]

In the study's non-tie comparison setting, GPT-4's judgments agreed with expert human judgments 85 percent of the time, while two sampled humans agreed 81 percent of the time. Those figures exclude ties and inconsistent judgments. In the setting that included ties and counted inconsistent order-swapped results as ties, GPT-4 pairwise judgments agreed with humans 66 percent of the time.^[2] The distinction is important when interpreting the often-repeated claim that an LLM judge matched human agreement.

The paper's single-answer GPT-4 scores included 2.61 for LLaMA-13B, 4.53 for Alpaca-13B, 6.00 for the full-data Vicuna-7B experiment, 6.39 for the full-data Vicuna-13B experiment, 7.94 for GPT-3.5, and 8.99 for GPT-4. A June 2023 LMSYS leaderboard update reported 7.12 for Vicuna-33B v1.3.^[2][10] These are results under a particular judge, prompt set, and model snapshot, not stable measures that should be compared indiscriminately with later leaderboard versions.

Chatbot Arena took a different approach. It presented two anonymous models with the same user prompt, asked the user to choose the preferred response, and revealed model identities after voting. The initial public leaderboard used an Elo rating.^[11] The 2024 technical report says the platform had collected more than 240,000 votes from about 90,000 users across more than 50 models by January 2024.^[12] Vicuna was among the arena's original models, but the platform and its methodology later encompassed many unrelated open-weight and proprietary systems.

FastChat and downstream use

FastChat is the Apache-2.0-licensed codebase released with Vicuna. It includes supervised fine-tuning code, command-line inference, a controller-and-worker system for distributed multi-model serving, web interfaces, an OpenAI-compatible REST API, and the MT-Bench evaluation pipeline. Its current repository describes FastChat as the release repository for both Vicuna and Chatbot Arena.^[6]

Vicuna also became a component in documented downstream research. The original LLaVA system connected a CLIP vision encoder to Vicuna and then performed visual instruction tuning. The LLaVA project explicitly identifies Vicuna as its language-model component.^[13] This is a concrete example of Vicuna's use as a research backbone. Broader claims that later chat models, APIs, or evaluation systems all descend from Vicuna require separate evidence and should not be inferred from this example.

Licensing and "open source" terminology

The original LMSYS post called Vicuna an open-source chatbot, but the project's components have different terms. FastChat's code is under the Apache License 2.0. The initial online demo was described as a non-commercial research preview subject to the original LLaMA license, OpenAI terms covering the source data, and ShareGPT privacy practices. LMSYS released code and weights but did not release the collected training dataset.^[1]

The Hugging Face model card for Vicuna-33B v1.3 labels its license non-commercial.^[14] Vicuna v1.5 model cards instead specify the Llama 2 Community License.^[7] That license permits many research and commercial uses but includes redistribution conditions, an acceptable-use policy, a restriction on using Llama materials or outputs to improve another large language model, and a separate licensing requirement for certain products with more than 700 million monthly active users at Llama 2's release.^[15]

The Open Source Initiative concluded that the Llama 2 license does not meet the Open Source Definition because it restricts some users and fields of use.^[16] For that reason, "open-weight" or "source-available" is more precise for Vicuna's model checkpoints. "Open source" accurately describes the Apache-licensed FastChat code and also reflects LMSYS's original terminology, but it should not be read as saying that every weight release uses an OSI-approved license.

Limitations and present status

LMSYS's original release identified weak reasoning and mathematics, factual errors, incorrect self-identification, and insufficient safety work for toxicity and bias. The public demo filtered user inputs with the OpenAI moderation API, which was a system-level safeguard rather than a property of the model weights.^[1] The later MT-Bench study likewise found Vicuna-13B notably behind GPT-3.5 and GPT-4 in reasoning, mathematics, and coding preference results.^[2]

Vicuna's training data also limits what can be independently audited. LMSYS documented high-level cleaning steps and aggregate counts, but did not publish the ShareGPT training corpus. Reproduction therefore requires a separately obtained dataset with similar formatting, not simply the released trainer and model weights.^[1][6]

The official v1.5 cards describe the intended users as researchers and hobbyists studying language models and chatbots.^[7] The checkpoints remain listed in the FastChat repository and on Hugging Face, making them usable for reproducing 2023 research and comparing early chat-tuning methods. The available primary sources do not establish that LMSYS formally discontinued Vicuna, that no one deploys it, or that FastChat is maintained primarily for Vicuna. The defensible current description is narrower: v1.5 remains the latest documented Vicuna release line, while FastChat, MT-Bench, and Chatbot Arena continued as broader projects.

References

1. ^Chiang, Wei-Lin, et al. "Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality." LMSYS, March 30, 2023. lmsys.org/...2023-03-30-vicuna
2. ^Zheng, Lianmin, et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023 Datasets and Benchmarks Track, arXiv:2306.05685. arxiv.org/...2306.05685
3. ^LMSYS. "Vicuna Weights." FastChat version notes. github.com/...vicuna_weights_version.md
4. ^Touvron, Hugo, et al. "LLaMA: Open and Efficient Foundation Language Models." arXiv:2302.13971, 2023. arxiv.org/...2302.13971
5. ^Taori, Rohan, et al. "Stanford Alpaca: An Instruction-following LLaMA Model." Stanford CRFM, 2023. github.com/...stanford_alpaca
6. ^LMSYS. "FastChat: An Open Platform for Training, Serving, and Evaluating Large Language Model Based Chatbots." GitHub repository. github.com/...FastChat
7. ^LMSYS. "Vicuna-13B v1.5 Model Card" and "Vicuna-13B v1.5-16K Model Card." Hugging Face. huggingface.co/...vicuna-13b-v1.5 ; huggingface.co/...vicuna-13b-v1.5-16k
8. ^Wang, Peiyi, et al. "Large Language Models are not Fair Evaluators." arXiv:2305.17926, 2023. arxiv.org/...2305.17926
9. ^Gudibande, Arnav, et al. "The False Promise of Imitating Proprietary LLMs." ICLR 2024, arXiv:2305.15717. arxiv.org/...2305.15717
10. ^Zheng, Lianmin, et al. "Chatbot Arena Leaderboard Week 8: Introducing MT-Bench and Vicuna-33B." LMSYS, June 22, 2023. lmsys.org/...2023-06-22-leaderboard
11. ^Zheng, Lianmin, et al. "Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings." LMSYS, May 3, 2023. lmsys.org/...2023-05-03-arena
12. ^Chiang, Wei-Lin, et al. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." ICML 2024, arXiv:2403.04132. arxiv.org/...2403.04132
13. ^Liu, Haotian, et al. "Visual Instruction Tuning." NeurIPS 2023, arXiv:2304.08485. arxiv.org/...2304.08485
14. ^LMSYS. "Vicuna-33B v1.3 Model Card." Hugging Face. huggingface.co/...vicuna-33b-v1.3
15. ^Meta. "Llama 2 Community License Agreement." July 18, 2023. ai.meta.com/...license
16. ^Maffulli, Stefano. "Meta's LLaMa License Is Not Open Source." Open Source Initiative, July 20, 2023. opensource.org/...ama-2-license-is-not-open-source