# BookCorpus

> Source: https://aiwiki.ai/wiki/bookcorpus
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: AI Ethics, Data & Datasets, Natural Language Processing
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "BookCorpus." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/bookcorpus
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

BookCorpus (also written BooksCorpus, and sometimes called the Toronto Book Corpus) is a text dataset built from free, self-published English-language ebooks scraped from the distribution platform Smashwords. It was introduced in 2015 by Yukun Zhu, Ryan Kiros, and colleagues at the University of Toronto and [MIT](https://aiwiki.ai/wiki/mit) for the paper "Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books", which reported a corpus of 11,038 books containing 984,846,357 words [1][3]. Despite its modest size by later standards (a few gigabytes of plain text), BookCorpus became one of the most consequential datasets in [natural language processing](https://aiwiki.ai/wiki/natural_language_processing): [OpenAI](https://aiwiki.ai/wiki/openai) pre-trained [GPT-1](https://aiwiki.ai/wiki/gpt-1) on it, and Google pre-trained [BERT](https://aiwiki.ai/wiki/bert) on BookCorpus plus English Wikipedia [4][5].

The dataset is equally significant for what went wrong with it. The original authors stopped distributing it, leaving a widely cited training corpus with no official copy. A 2021 retrospective datasheet by Jack Bandy and Nicholas Vincent found that it contained only 7,185 unique books rather than 11,038, that many books carried explicit copyright notices forbidding redistribution, and that its genre balance skewed heavily toward romance [3]. BookCorpus is now a standard case study in dataset documentation and consent debates. It should not be confused with Books3, a separate and much larger collection of pirated published books that became central to AI copyright litigation; the two are related only in that both are book datasets used for language model training [9][12].

## Origins

BookCorpus was assembled to support research on aligning scenes in films with passages in the novels they adapt. Zhu, Kiros, and co-authors needed a large amount of contiguous narrative text to train a sentence encoder in an unsupervised way, and described their collection method in a single paragraph: "In order to train our sentence similarity model we collected a corpus of 11,038 books from the web. These are free books written by yet unpublished authors," filtered to books longer than 20,000 words, across 16 genres [1][3]. The same corpus trained the skip-thought vectors sentence embedding model published by the same group in June 2015, an influential precursor to modern sentence representation methods [2]. The alignment paper was presented at the International Conference on Computer Vision in December 2015, and its acknowledgments credit funding from NSERC, CIFAR, Samsung, Google, and the US Office of Naval Research [1][3].

The books came from Smashwords, a self-publishing platform launched in 2008 that by 2014 hosted 336,400 titles from 101,300 authors [3]. BookCorpus took only books offered at a price of zero and longer than 20,000 words, a threshold that matches a filter in the Smashwords interface for books "Over 20K words" [3]. Bandy and Vincent characterize the result as a non-probabilistic convenience sample covering roughly 2 to 3 percent of what Smashwords then hosted [3].

## What the dataset contains

The original paper reported six summary statistics [1][3]:

| Statistic | Value |
|---|---|
| Books | 11,038 |
| Sentences | 74,004,228 |
| Words | 984,846,357 |
| Unique words | 1,316,420 |
| Mean words per sentence | 13 |
| Median words per sentence | 11 |

As distributed, BookCorpus consisted of one plain-text file per book, organized into 16 genre folders, plus two large files with one sentence per line. Each file held the full text extracted from an ebook, often including front matter and licensing boilerplate [3]. There were no labels; the only label-like attribute was the genre assigned on Smashwords [3]. Models consuming the corpus applied their own preprocessing, for example GPT-1 cleaned the text with the ftfy library, tokenized it with spaCy, and encoded it with [byte pair encoding](https://aiwiki.ai/wiki/byte_pair_encoding) [4].

## Role in language model pre-training

BookCorpus's importance grew after its creators' original use case faded. Its long stretches of continuous prose made it a natural fit for the generative pre-training paradigm. The GPT-1 paper explains the choice directly: "We use the BooksCorpus dataset for training the language model. It contains over 7,000 unique unpublished books from a variety of genres including Adventure, Fantasy, and Romance. Crucially, it contains long stretches of contiguous text, which allows the generative model to learn to condition on long-range information" [4]. That property distinguished it from shuffled sentence-level corpora such as the 1B Word Benchmark [4].

BERT (2018) pre-trained on "the BooksCorpus (800M words)" together with English Wikipedia (2,500M words), and that recipe propagated through BERT's descendants: [RoBERTa](https://aiwiki.ai/wiki/roberta), [XLNet](https://aiwiki.ai/wiki/xlnet), [ALBERT](https://aiwiki.ai/wiki/albert), and [DistilBERT](https://aiwiki.ai/wiki/distilbert) all list BookCorpus-derived text in their pre-training data [3][5][7]. Bandy and Vincent counted more than thirty influential [language models](https://aiwiki.ai/wiki/language_model) trained or fine-tuned on the dataset as listed on its [Hugging Face](https://aiwiki.ai/wiki/hugging_face) dataset card [3]. Later OpenAI models moved to larger, undisclosed book collections: the [GPT-3](https://aiwiki.ai/wiki/gpt-3) paper lists "two internet-based books corpora" called simply Books1 (12 billion tokens) and Books2 (55 billion tokens) without identifying their contents, a naming choice that has fed persistent confusion between BookCorpus and other book datasets [10].

Public attention arrived in September 2016, when The Guardian reported that [Google Brain](https://aiwiki.ai/wiki/google_brain) researchers, including Oriol Vinyals and Andrew M. Dai, had used the corpus for conversational language research. Novelist Rebecca Forster, whose free thriller Hostile Witness was in the corpus, said no one had asked permission to use her novel as training material: "it didn't even occur to me that a machine could read my book." The Authors Guild condemned the use as "blatantly commercial use of expressive authorship" [6].

## Distribution history and replications

The corpus was originally downloadable in full from a University of Toronto project page [6]. At some point after publication the authors withdrew it; by 2021 the page instead read, "Please visit smashwords.com to collect your own version of BookCorpus" [3]. There has been no official version since. Bandy and Vincent obtained the original files only through a security hole in the defunct distribution site, which they reported to the authors [3].

The vacuum produced replications. "Homemade BookCorpus" scripts let researchers re-scrape free Smashwords books themselves [3]. The best-known replica is BookCorpusOpen, assembled by programmer Shawn Presser and hosted by The Eye, containing 17,868 book texts from a list of 18,060 Smashwords URLs; it has circulated under names including BookCorpusNew, Books1, and OpenBookCorpus [3][8]. [EleutherAI](https://aiwiki.ai/wiki/eleutherai) included an expanded variant in [The Pile](https://aiwiki.ai/wiki/the_pile) under the name BookCorpus2 (6.30 GiB) [9]. BookCorpusOpen's listing on Hugging Face now sits in a "defunct-datasets" organization with the notice that the dataset "is defunct and no longer accessible due to unavailability of the source data" [8]. A copy of the original BookCorpus nevertheless remains downloadable from Hugging Face as of 2026, with its license listed as "unknown" and a card that summarizes the documented consent and copyright problems [7].

## The retrospective datasheet

In May 2021, Bandy and Vincent published "Addressing 'Documentation Debt' in Machine Learning Research: A Retrospective Datasheet for BookCorpus", applying the datasheets-for-datasets framework to a corpus that had been used widely but documented in a few sentences [3]. Comparing the original files, BookCorpusOpen, and a fresh scrape of Smashwords metadata ("Smashwords21", 411,826 books), they found:

- **Duplication.** Only 7,185 of the 11,038 books were unique. 2,930 books occurred more than once: 2,101 appeared twice, 741 three times, 82 four times, and 6 five times [3].
- **Missing and truncated text.** 98 book files were empty, 655 were shorter than the paper's own 20,000-word floor, and 291 were under 10,000 words, suggesting significant truncation [3]. The per-book files summed to about 812 million words, roughly 170 million short of the 985 million in the sentence-level files [3].
- **Copyright restrictions.** Thousands of files retained licensing boilerplate. The sentence "if you're reading this book and did not purchase it, or it was not purchased for your use only, then please return to smashwords.com and purchase your own copy" appeared 788 times, and the authors counted hundreds of further instances of explicit no-redistribution and no-reproduction language across other files [3]. At least 438 books in the corpus were no longer free on Smashwords by 2021; in a subset of 2,680 books matched to current listings, buying the 406 that now cost money would have totaled $1,182.21 as of April 2021 [3].
- **Genre skew.** Romance made up 26.1 percent of BookCorpus versus 16.0 percent of all publicly listed Smashwords books; a dedicated vampire subgenre accounted for 5.4 percent of the corpus. The datasheet describes the corpus as not a significant source of nonfiction [3].
- **Concentration and privacy.** An estimated 3,490 authors wrote the corpus, and among free Smashwords books the top 10 percent of authors by word count supplied 59 percent of all words. Many files contained authors' personal email addresses [3].
- **Content flags.** 537,878 unique sentences contained at least one term from a common list of profane or obscene words, spanning innocuous profanity, pornographic dialogue, and slurs; the authors flagged this, along with skews in religious representation, for further study [3].

The paper's title borrows the "documentation debt" framing from Emily Bender, [Timnit Gebru](https://aiwiki.ai/wiki/timnit_gebru), and colleagues: influential models were being trained on corpora nobody could fully describe or replicate [3]. The GPT-1 paper had already described the corpus as containing "over 7,000 unique unpublished books", a figure consistent with the 7,185 unique titles Bandy and Vincent counted three years later [3][4].

## Consent and copyright

BookCorpus occupies an odd legal position. Every book in it was, at collection time, offered for free by its author, but free to read is not free to redistribute or to license onward. Many of the books reserved all rights and were "licensed for your personal enjoyment only" [3][6]. Authors were not asked, were not notified, and had no mechanism to remove their books after the fact; books that later went paid or changed license remained in circulating copies [3]. Bandy and Vincent conclude that researchers should use the dataset with caution for any task, citing the copyright issues, duplicates, and sampling skews [3]. Because the corpus's authors were mostly self-published writers rather than commercial publishers, these concerns produced discussion rather than lawsuits, which is one of several ways BookCorpus differs from the dataset it is most often confused with.

## Not the same dataset as Books3

Books3 is frequently conflated with BookCorpus, including in press coverage, but the two are different in origin, scale, and legal history:

| | BookCorpus (2015) | Books3 (2020) |
|---|---|---|
| Assembled by | Zhu, Kiros et al. (University of Toronto/MIT) | Shawn Presser (independent) |
| Source | Free self-published ebooks on Smashwords | Bibliotik, a private "shadow library" tracker of pirated published books |
| Size | 11,038 files, 7,185 unique books, ~985M words | 196,640 books, ~37 GB plain text [12] |
| Distribution | Author website (withdrawn); unofficial mirrors | The Eye; included in The Pile (100.96 GiB component) [9] |
| Known model uses | GPT-1, BERT and variants [3][4][5] | [LLaMA](https://aiwiki.ai/wiki/llama), models trained on The Pile [9][11] |
| Legal status | Copyright concerns documented; no litigation centered on it | DMCA takedown 2023; central to author lawsuits [12][13] |

Presser released Books3 in late 2020 as part of the effort that became The Pile, whose authors describe it plainly as "a dataset of books derived from a copy of the contents of the Bibliotik private tracker" [9][12]. [Meta AI](https://aiwiki.ai/wiki/meta_ai) disclosed in the LLaMA paper that its book data came from Project Gutenberg and "the Books3 section of ThePile" [11]. That disclosure fed directly into [Kadrey v. Meta](https://aiwiki.ai/wiki/kadrey_v_meta), a class action filed by authors in the Northern District of California on July 7, 2023, over the books used to train LLaMA [13]. In August 2023 the Danish anti-piracy group Rights Alliance had Books3 taken offline from The Eye via a DMCA takedown notice, though copies persisted in Pile torrents and mirrors [12].

The wider book-piracy litigation continued to escalate through 2025 and 2026. In [Bartz v. Anthropic](https://aiwiki.ai/wiki/bartz_v_anthropic) (N.D. Cal., No. 3:24-cv-05417), [Anthropic](https://aiwiki.ai/wiki/anthropic) agreed to a $1.5 billion class settlement with authors over roughly 500,000 books it had downloaded from the LibGen and PiLiMi pirate libraries, with claims due by March 30, 2026 [14][15]. None of this litigation concerns BookCorpus itself, but the underlying question, whether AI developers may train on books without permission, is the one BookCorpus raised in miniature a decade earlier [3][6]. See [AI copyright](https://aiwiki.ai/wiki/ai_copyright) for the broader landscape.

## Legacy

BookCorpus's paragraph-long documentation, quiet withdrawal, and posthumous audit made it a reference point for arguments that datasets need the same rigor as models. The Bandy and Vincent datasheet became a widely cited example of a retrospective dataset audit, and the case features regularly in work on training data transparency and [AI ethics](https://aiwiki.ai/wiki/ai_ethics) [3]. The failures it exposed (train-time duplication, undocumented provenance, unresolved rights) are exactly what later open corpora tried to engineer away, from The Pile's documented subsets to [C4](https://aiwiki.ai/wiki/c4_dataset)'s reproducible [Common Crawl](https://aiwiki.ai/wiki/common_crawl) filtering and openly documented efforts like [FineWeb](https://aiwiki.ai/wiki/fineweb), [Dolma](https://aiwiki.ai/wiki/dolma), and the openly licensed [Common Pile](https://aiwiki.ai/wiki/common_pile) [3][9]. Meanwhile the corpus itself lives on in the weights of BERT-era models and in mirrors that no one officially maintains: a foundational dataset that, strictly speaking, no longer exists anywhere in an authorized form [3][7][8].

## See also

- [The Pile](https://aiwiki.ai/wiki/the_pile)
- [C4 dataset](https://aiwiki.ai/wiki/c4_dataset)
- [Common Crawl](https://aiwiki.ai/wiki/common_crawl)
- [AI copyright](https://aiwiki.ai/wiki/ai_copyright)
- [Kadrey v. Meta](https://aiwiki.ai/wiki/kadrey_v_meta)
- [Open Images](https://aiwiki.ai/wiki/open_images)

## References

1. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., Fidler, S. "Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books." arXiv, June 22, 2015. https://arxiv.org/abs/1506.06724
2. Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R., Torralba, A., Urtasun, R., Fidler, S. "Skip-Thought Vectors." arXiv, June 22, 2015. https://arxiv.org/abs/1506.06726
3. Bandy, J., Vincent, N. "Addressing 'Documentation Debt' in Machine Learning Research: A Retrospective Datasheet for BookCorpus." arXiv, May 11, 2021. https://arxiv.org/abs/2105.05241
4. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. "Improving Language Understanding by Generative Pre-Training." OpenAI, 2018. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
5. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv, October 11, 2018. https://arxiv.org/abs/1810.04805
6. Lea, R. "Google swallows 11,000 novels to improve AI's conversation." The Guardian, September 28, 2016. https://www.theguardian.com/books/2016/sep/28/google-swallows-11000-novels-to-improve-ais-conversation
7. Hugging Face. "bookcorpus" dataset card. https://huggingface.co/datasets/bookcorpus/bookcorpus
8. Hugging Face. "bookcorpusopen" dataset card (defunct-datasets). https://huggingface.co/datasets/defunct-datasets/bookcorpusopen
9. Gao, L., Biderman, S., Black, S., et al. "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv, December 31, 2020. https://arxiv.org/abs/2101.00027
10. Brown, T., et al. "Language Models are Few-Shot Learners." arXiv, May 2020. https://arxiv.org/abs/2005.14165
11. Touvron, H., Lavril, T., Izacard, G., Martinet, X., et al. "LLaMA: Open and Efficient Foundation Language Models." arXiv, February 27, 2023. https://arxiv.org/abs/2302.13971
12. TorrentFreak. "Anti-Piracy Group Takes Prominent AI Training Dataset 'Books3' Offline." August 16, 2023. https://torrentfreak.com/anti-piracy-group-takes-prominent-ai-training-dataset-books3-offline-230816/
13. Joseph Saveri Law Firm. "Meta Platforms, Inc. AI Language Model Litigation (Kadrey v. Meta)." https://www.saverilawfirm.com/meta-language-model-litigation
14. The Authors Guild. "Anthropic Settlement FAQs" (Bartz v. Anthropic). https://authorsguild.org/advocacy/artificial-intelligence/anthropic-settlement-faq/
15. JND Legal Administration. "Anthropic Copyright Settlement" (Bartz, et al. v. Anthropic PBC, No. 3:24-cv-05417). https://www.anthropiccopyrightsettlement.com/