AI Wiki

Explore AI Wiki

Citation and evidence

Fish Audio

14 min full readUpdated 23 references

Report a problem with this article

More

Use this article

Raw MarkdownExplore connections

Improve this page

Suggest editRevision history

Topics

AI CompaniesOpen Source AISpeech & Audio AIVoice AI

Cite this article

Fish Audio is an American voice AI company that develops text-to-speech (TTS) and voice cloning models and runs the fish.audio platform for creators, developers and enterprises. The company is legally named Hanabi AI Inc. and is based in Palo Alto, California.[1][2] It grew out of Fish Speech, an open-source TTS project started by Shijia Liao, and its later model families include OpenAudio S1, Fish Audio S2 and Drama 3.[1][3][5] In July 2026 the company announced a US$52 million seed round co-led by Coreline Ventures and Capital Today.[1][3] On 13 October 2026 it released Drama 3, an open-weight speech generation model built on a 35-billion-parameter mixture-of-experts backbone with about 3 billion parameters active per token.[5][6]

Fish Audio logo
Fish Audio logo, from the official website fish.audio. Trademark of its owner.

Overview

FieldDetails
Legal nameHanabi AI Inc.[2]
HeadquartersPalo Alto, California[1]
Co-foundersRissa Cao (chief executive officer), Shijia Liao (chief scientist)[1][2]; company blog author pages also describe Jiahua Liu (chief technology officer) and Zhizhuo Zhou as co-founders[5][13]
Main productsfish.audio web platform and app, Fish Audio API, open-weight Fish Speech, S2 and Drama 3 models[5][16][20]
FundingUS$52 million seed round (announced July 2026)[1][3]
Reported scale (company figures)More than 8 million users and US$21 million in annual recurring revenue (July 2026); "15M+ users" (October 2026)[1][3][4]

Expanded article table

History

Fish Speech

Fish Audio began as a personal project by Shijia Liao. According to the company and TechCrunch, Liao, then a researcher at Nvidia (the company's account says he worked on video), was dissatisfied with how flat synthetic voices sounded, trained a voice model on a single GPU and released it as open source.[1][3] The company's own telling says the GPU was a single 4090 in his bedroom and that the work was prompted by the lifeless voices he heard while watching VTuber streams.[3] The resulting project, Fish Speech, is hosted at github.com/fishaudio/fish-speech; TechCrunch reported in July 2026 that the repository had more than 31,000 stars and was used by indie developers, video game designers and creators.[1][20]

Fish Audio's Hugging Face organization lists a Fish Speech 1 checkpoint created in April 2024 and Fish Speech 1.2, 1.4 and 1.5 checkpoints created between July and November 2024.[21] The system was described in an arXiv paper submitted on 2 November 2024, "Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis," by Liao and six co-authors.[9] The paper presents a serial fast-slow "Dual Autoregressive" (Dual-AR) architecture over grouped finite scalar vector quantization (GFSQ) codes, uses a large language model for linguistic feature extraction instead of grapheme-to-phoneme conversion, and introduces a vocoder the authors call FF-GAN with near 100% codebook utilization.[9] A company summary of the paper says the model was trained on 720,000 hours of multilingual audio.[10]

OpenAudio S1

On 3 June 2025, Hanabi AI announced OpenAudio S1 in a Business Wire press release, calling it "the world's first AI voice actor"; the model was offered in open beta on fish.audio.[11] The release, which identified Liao as "founder and CEO of Hanabi AI," described an end-to-end architecture with 4 billion parameters and control over delivery through markers such as "(angry)" or "(voice quivering)" as well as freer instructions like "(confident but hiding fear)".[11][12] Speech Technology magazine reported that S1 handled 11 languages.[12] The "first AI voice actor" label is the company's marketing claim.

A later company post, titled "Launching Fish Audio S1," gives further detail: S1 (4B parameters) ran on the hosted platform, while a distilled S1-mini (0.5B) was published on Hugging Face; both were trained with online reinforcement learning from human feedback using GRPO, and S1 was trained on more than 2 million hours of audio using the Qwen3 architecture as its backbone.[13] The post reports a word error rate of 0.008 and character error rate of 0.004 on Seed-TTS Eval, measured with GPT-4o-based transcription (a company-run evaluation).[13] The S1-mini checkpoint was uploaded on 31 May 2025 under the CC BY-NC-SA 4.0 license.[14]

Fish Audio S2 and S2.1 Pro

Fish Audio released S2 with open weights on 9 March 2026, together with a technical report on arXiv, fine-tuning code and an SGLang-based inference engine.[15][16] S2 supports multi-speaker, multi-turn generation and natural-language instructions written inline as free-form tags such as "[laugh]" or "[whispers]".[16] The company describes a Dual-AR design with about 4 billion parameters along the time axis and 400 million along the depth axis, an RVQ-based audio codec with 10 codebooks at roughly 21 Hz, and training on more than 10 million hours of audio in about 80 languages.[16] Self-reported results include word error rates of 0.54% (Chinese) and 0.99% (English) on Seed-TTS Eval and an 81.88% win rate against a gpt-4o-mini-tts baseline on EmergentTTS-Eval.[16] The s2-pro weights on Hugging Face are released under the Fish Audio Research License; the model card says research and non-commercial use is free of charge and commercial use requires a separate license from Fish Audio.[17]

On 23 June 2026 the company made S2.1 Pro, which it called its most advanced model at the time, available as a free API tier (model string "s2.1-pro-free") subject to a fair-use policy.[18] Fish Audio said S2.1 Pro won 61% of head-to-head listening comparisons against S2 Pro and cut single-request time to first audio to about 70 ms.[18] TechCrunch reported in July 2026 that Fish Audio had open-sourced three of its speech-generation models but that S2.1 Pro was available only through the company's API.[1]

Seed funding

TechCrunch and SiliconANGLE reported on 28 July 2026 that Fish Audio had raised US$52 million in a seed round led by Coreline Ventures and Capital Today.[1][2] The company's announcement listed participation from 359 Capital, Play Time, HF0, 645 Ventures, Parable, Carya Venture Partners, Alphalist Partners and angel investors; TechCrunch's list also named Bayhouse Ventures.[1][3]

The company's announcement, written by Cao, said the round came on Fish Audio's first anniversary and claimed that in that year the company had shipped five audio models (four text-to-speech, one speech-to-text), grown from 3 to 22 employees, reached US$21 million in annual recurring revenue and passed 8 million users and 2 million voice models in its community library.[3] It said enterprise customers and developers together made up two-thirds of revenue.[3] Cao told TechCrunch that the company had run efficiently on open source and creator plans but sought capital to build more advanced models and serve enterprises.[1] Stated plans included an audio understanding language model, a speech-to-speech model, more developer tooling and an expanded enterprise sales team, with integrations through partners such as LiveKit and Retell AI.[1][2][3] To mark the round, S2.1 Pro was offered free to developers via the API through the end of August 2026.[3] Fish Audio later extended the free S2.1 Pro API access through 30 November 2026.[18]

On 13 October 2026, the company's X account restated the round and gave a figure of "15M+ users" while announcing Drama 3.[4]

Drama 3

Drama 3 (full name Fish Audio Drama 3) was released on 13 October 2026 on the fish.audio web app, the Fish Audio iOS and Android apps, the API and as open weights on Hugging Face.[5][7] The company's X post called it "the most emotive and controllable TTS model in the world," and its launch blog post described it as Fish Audio's most emotive text-to-speech model.[4][5] A technical report by the Fish Audio Team (16 listed authors, including Liao and Liu) was posted to arXiv the same day.[6]

Architecture

ComponentDescription (per Fish Audio)
Backbone35B-parameter mixture-of-experts language model, about 3B active per token; each layer routes a token to 8 of 256 experts plus one shared expert[6]
InitializationLanguage model of Qwen3.5-35B-A3B[6]
Attention40 layers in blocks of three Gated DeltaNet (linear attention) layers and one gated full-attention layer; only one layer in four keeps a key-value cache[6]
Audio decoderNew codebook multi-token prediction (MTP) decoder: four shared layers followed by nine per-codebook layers, replacing S2's per-frame "Fast AR" module[6]
Audio tokenizerReused unchanged from S2; built on the Descript Audio Codec, 44.1 kHz audio, ten codebooks (one semantic, nine residual) at about 21.5 frames per second, fully causal[6]
Codebook feedbackOnly the semantic code and first acoustic code are fed back into the backbone ("partial codebook fusion"), compared with all ten in S2[6]
SpeakersUp to 16 characters per text via speaker tags[7]
WeightsReleased checkpoint lists about 35.9 billion parameters, with mixture-of-experts experts in FP8 and other weights in BF16[7]

Expanded article table

Training used three pre-training stages over several hundred billion tokens of transcribed speech and text documents, covering the roughly 80 languages and dialects of S2, followed by instruction fine-tuning and reinforcement learning.[6] The reinforcement learning stage uses GRPO without a KL penalty, on large batches, with a single learned reward model of human preference and no rule-based rewards for word error rate or speaker similarity; the authors argue that such proxy rewards penalize intended laughs, pauses or whispers.[6] The launch post gives the batch as 1,536 prompts with 8 takes each per step.[5]

Capabilities

The model clones voices from one or more reference clips, follows natural-language direction written in square brackets, performs multi-speaker dialogue and multi-character scenes in one pass, and synthesizes long-form or streaming audio.[6] The technical report lists multi-turn revision (redoing a take from a note such as "[softer and warmer versus previous]"), scene dubbing, editing by infilling and streaming text input as capabilities new in Drama 3.[6] Paired tags such as <emphasis>...</emphasis> direct only the enclosed words, and the company says direction can be written in a different language from the line being spoken.[5] Fish Audio says Drama 3 speaks 83 languages.[5] At launch, Fish Audio said interactive editing and audio inpainting tools were in invite-only testing, with an editing API to follow.[5] The company's announcement video featured Uncle Roger directing an AI clone of his own voice, which Fish Audio says was made and used with his permission.[5]

Evaluation

Results in the technical report are self-reported. On Seed-TTS-Eval, reinforcement learning lowered the first-take error rate from 1.66 to 1.00 CER on the Chinese test set and from 3.49 to 2.20 WER on the English set, relative to the supervised fine-tuning checkpoint.[6] The report says Drama 3 had the lowest error rate of the compared systems in Chinese, Dutch, English, German and Turkish on the MiniMax Multilingual Testset, and a lower error rate than Cartesia's Sonic-3 infill API in English and Chinese on the Ming-Freeform-Audio-Edit benchmark.[6] It also claims that, to the authors' knowledge, Drama 3 is the largest open-weight model dedicated to speech generation by total parameters.[6]

On Artificial Analysis's independent text-to-speech benchmarks, "Fish Audio Drama 3 Preview" had a Pronunciation Robustness pass rate of 93.2%, the highest listed, ahead of ElevenLabs' Eleven v4 (91.7%) and Fish Audio's own S2.1 Pro (88.1%), as of 15 October 2026.[19] On the same site's arena preference ranking it had an Elo of 1,242, below models such as Eleven v4 (1,326) and Cartesia's Sonic 3.6 (1,283).[19]

Inference metric (Fish Audio figures)Value
Single-stream decoding speed on one NVIDIA H200271 frames per second (177 for S2)[6]
Time to first audio, median, concurrency 1 (H200, SGLang-Omni)88.9 ms[7]
Mean real-time factor at concurrency 640.801[7]

Expanded article table

License and availability

Drama 3 is released under the Fish Audio Community License Agreement, version 1.0, dated 13 October 2026.[8] It permits free research and non-commercial use, and commercial use only while the licensee and its affiliates have annual revenue below US$500,000; above that threshold, or to offer a hosted service such as a TTS, voice-cloning or inference API to third parties, a separate license from Fish Audio is required.[5][8] The launch post says the license also bans cloning a real person's voice without consent, and that S2 remains under the Fish Audio Research License.[5] Fish Audio's launch FAQ describes Drama 3 as "open-weight rather than open source", although the technical report calls it "an open-source speech generation model".[5][6] The license defines "Fish Audio" as 39 AI, Inc. and its affiliates.[8] Inference runs on SGLang-Omni from launch.[7]

In the API, Drama 3 is selected with a "drama-3" model header on the same endpoint as S2.1 Pro.[5] Fish Audio set a launch price of US$10 per million characters from 13 October to 12 November 2026, which it described as 80% off a US$50 list price.[5]

Self-hosted inference

Fish Audio's installation documentation lists at least 48 GB of GPU memory for Drama 3's FP8 checkpoint and requires an NVIDIA GPU. Its supported system list is Linux or Windows Subsystem for Linux (WSL).[22] The Drama 3 runtime does not support CPU, Apple MPS, AMD ROCm or the --compile optimization. Drama 3 and S2-Pro use the same command-line tools and API server, which selects the model from the checkpoint's config.json; S2-Pro's wider hardware support does not apply to Drama 3.[22][23]

Products and customers

The fish.audio platform offers paid monthly creator and team plans with set generation minutes and voice cloning, plus an enterprise API and platform.[1] TechCrunch reported that organizations including HeyGen and Sanas used the enterprise product.[1] SiliconANGLE reported that Fish Audio offers on-premises deployment, zero-data-retention options and HIPAA-compliant configurations for regulated industries, and that the platform advertises voice cloning from a five-second clip and support for 83 languages.[2]

Fish Audio has built its voice library partly from user-submitted voices and compensates users whose voices are used.[1] TechCrunch reported in July 2026 that some creators had alleged their voices were uploaded to the platform without consent, and that the company's DMCA takedown process had been slow.[1] Cao said the company had since automated takedowns, so that a creator can submit a short voice sample or a contract proving ownership and have the voice removed in under three minutes; TechCrunch noted this does not stop a voice from being uploaded and used until the owner discovers it and files a request.[1] The Drama 3 license prohibits cloning a real person's voice without consent.[5]

See also

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19Ivan Mehta, "Fish Audio raises $52M seed to build AI voice models for creators and enterprises," TechCrunch, 28 July 2026. techcrunch.com/...els-for-creators-and-enterprises
  2. ^1 ^2 ^3 ^4 ^5 ^6Mike Wheatley, "Fish Audio makes a splash after raising $52M seed funding for AI voices," SiliconANGLE, 28 July 2026. siliconangle.com/...ing-52m-seed-funding-ai-voices
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Rissa Cao, "5 Models, 22 People, 1 Year" (seed funding announcement), Fish Audio Blog, July 2026. fish.audio/...fish-audio-52m-seed-funding
  4. ^1 ^2 ^3Fish Audio (@FishAudio), "$52M seed round. 15M+ users. ... Introducing Drama 3," X, 13 October 2026. x.com/...2110038429199450126
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Shijia Liao, Rissa Cao and Jiahua Liu, "Fish Audio Drama 3: The Most Emotive Text-to-Speech Model We've Built," Fish Audio Blog, 13 October 2026. fish.audio/...fish-audio-drama-3
  6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17Fish Audio Team (Dawei Han et al.), "Fish Audio Drama 3 Technical Report," arXiv:2610.18010, 13 October 2026 (v2 14 October 2026). arxiv.org/...2610.18010
  7. ^1 ^2 ^3 ^4 ^5 ^6"fishaudio/drama-3" model card, Hugging Face. Accessed 15 October 2026. huggingface.co/...drama-3
  8. ^1 ^2 ^3"Fish Audio Community License Agreement," version 1.0, 13 October 2026. huggingface.co/...LICENSE
  9. ^1 ^2Shijia Liao et al., "Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis," arXiv:2411.01156, 2 November 2024. arxiv.org/...2411.01156
  10. ^Shijia Liao, "Introducing Fish-Speech: A Next-Generation Multilingual TTS," Fish Audio Blog. fish.audio/...introducing-fish-speech
  11. ^1 ^2Hanabi AI Inc., "Hanabi AI Launches OpenAudio S1: The World's First AI Voice Actor for Real-Time Emotional Control," Business Wire, 3 June 2025. businesswire.com/...en
  12. ^1 ^2"Hanabi AI Launches OpenAudio S1," Speech Technology, 4 June 2025. speechtechmag.com/...-Launches-OpenAudio-S1-169838
  13. ^1 ^2 ^3Zhizhuo Zhou, "Launching Fish Audio S1: A Frontier Text-to-Speech Audio Foundation Model," Fish Audio Blog. fish.audio/...introducing-s1
  14. ^"fishaudio/s1-mini" model card, Hugging Face. huggingface.co/...s1-mini
  15. ^Shijia Liao et al., "Fish Audio S2 Technical Report," arXiv:2603.08823, 9 March 2026. arxiv.org/...2603.08823
  16. ^1 ^2 ^3 ^4 ^5Shijia Liao, "Fish Audio Open-Sources S2: Fine-Grained Control Meets Production Streaming," Fish Audio Blog, 9 March 2026. fish.audio/...fish-audio-open-sources-s2
  17. ^"fishaudio/s2-pro" model card, Hugging Face. huggingface.co/...s2-pro
  18. ^1 ^2 ^3Shijia Liao, "Fish Audio S2.1 Pro: Free Text-to-Speech API for Developers," Fish Audio Blog, 23 June 2026. fish.audio/...s2-1-pro-free-api
  19. ^1 ^2"Best Provider Voice Text to Speech (TTS) Models," Artificial Analysis. Accessed 15 October 2026. artificialanalysis.ai/text-to-speech
  20. ^1 ^2"fishaudio/fish-speech," GitHub. github.com/...fish-speech
  21. ^"fishaudio" organization models, Hugging Face. Accessed 15 October 2026. huggingface.co/fishaudio
  22. ^1 ^2Fish Audio, "Installation," Fish Speech documentation. Accessed 16 October 2026. speech.fish.audio/install
  23. ^Fish Audio, "Inference," Fish Speech documentation. Accessed 16 October 2026. speech.fish.audio/inference

Improve this article

Add missing citations, update stale details, or suggest a clearer explanation. Every suggestion is reviewed for sourcing before it goes live.

3 revisions · v3 · 2,763 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Cite this page: AI Wiki. "Fish Audio." aiwiki.ai, updated 28 Oct 2026. CC BY 4.0. https://aiwiki.ai/wiki/fish_audio

Suggest edit

Where to go next

Topics:AI CompaniesOpen Source AISpeech & Audio AIVoice AI