Donut (Model)

Donut (Document understanding transformer) is an OCR-free visual document understanding model introduced by researchers at NAVER CLOVA in the paper "OCR-free Document Understanding Transformer," first posted to arXiv in November 2021 and published at ECCV 2022 ^[1][2]. Instead of running a separate optical character recognition engine and then feeding the recognized text into a language model, Donut maps a raw document image directly to structured output: a Swin Transformer encoder converts the page into embeddings, and a BART-style text decoder generates a token sequence that converts one-to-one into JSON ^[2]. The authors describe it as, to the best of their knowledge, the first visual document understanding (VDU) method based on an OCR-free Transformer trained end to end ^[2].

The model was accompanied by SynthDoG, a synthetic document generator that let the team pretrain Donut in English, Chinese, Japanese, and Korean without needing large real-document corpora in each language ^[2]. Code, pretrained weights, and the synthetic data were released under the MIT license on GitHub, and the checkpoints are hosted on Hugging Face under the naver-clova-ix organization, with naver-clova-ix/donut-base as the main pretrained model ^[3][4]. Donut became a widely used open document AI model and an architectural template for later OCR-free document readers such as Meta's Nougat ^[7].

Background

At the time of Donut's development, the dominant approach to document understanding tasks (classifying scanned pages, extracting fields from receipts and invoices, answering questions about forms) was a two-stage pipeline: an off-the-shelf OCR engine read the text, and a downstream model such as LayoutLM combined the OCR tokens, their bounding-box coordinates, and sometimes image features to do the actual understanding ^[2]. The Donut paper identifies three problems with this design: OCR inference is computationally expensive; off-the-shelf OCR engines generalize poorly across languages and document types, while training a custom OCR model demands large datasets and cost; and OCR errors propagate into every downstream prediction ^[1][2]. The error-propagation problem is worst for languages with complex character sets such as Korean and Chinese, where OCR quality is comparatively low and systems often bolt on a post-OCR correction module, adding further size and maintenance cost ^[2].

Donut's answer was to remove the OCR stage entirely and model a direct mapping from pixels to the desired structured output. The name is a contraction of "Document understanding transformer," and the project uses a donut emoji as its logo ^[2][3].

The work came out of NAVER CLOVA, the AI division of the Korean internet company NAVER. The ECCV paper lists ten authors: Geewook Kim (corresponding author), Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. At publication, Kim, Nam, and Seunghyun Park were listed at NAVER CLOVA and Yun and Han at NAVER AI Lab; Hong (Upstage), Moonbin Yim (NAVER Search), Jinyoung Park (Tmax), Jinyeong Yim (Google), and Hwang (LBox) are marked as having done the work while at NAVER CLOVA ^[2].

Architecture

Donut is a plain encoder-decoder Transformer with no OCR-specific components ^[2].

Scroll sideways for more →

The encoder splits the input image into non-overlapping patches, processes them through shifted-window self-attention blocks with patch merging between stages, and hands the final feature map to the decoder ^[2]. The authors tried CNN backbones such as EfficientNetV2 as well and chose Swin because it performed best in their document-parsing studies ^[2].

The decoder generates output autoregressively. Following GPT-3, the task is specified by a prompt: a special token sequence such as <s_rvlcdip> for classification, <s_cord-v2> for receipt parsing, or <s_docvqa><s_question>...</s_question><s_answer> for question answering is fed to the decoder, which then completes the answer ^[2][5]. Training uses teacher forcing. All downstream tasks are cast uniformly as JSON prediction: the model emits paired special tokens like [START_name] and [END_name] around each field, so the token sequence converts to JSON with simple regular expressions, and a field whose closing token is missing is treated as lost rather than crashing the parse ^[2]. Because Donut is trained without any localization supervision, the paper also shows that the decoder's cross-attention maps land on the text regions being read, which can serve as an auxiliary text-localization signal ^[2].

Pretraining and SynthDoG

Donut's pretraining objective is deliberately simple: read all the text in the image in reading order (roughly top-left to bottom-right), minimizing the cross-entropy loss of next-token prediction conditioned jointly on the image and the preceding text ^[2]. The paper calls this a pseudo-OCR task and describes the resulting model as a visual language model over document images ^[2]. In the pretraining ablations, this text-reading task beat alternatives such as image captioning for downstream document tasks ^[2].

The visual corpus combines real and synthetic data. The real part is IIT-CDIP, a set of 11 million scanned English document images, with pseudo text labels produced by NAVER's commercial CLOVA OCR API ^[2]. Because such corpora barely exist for other languages, the team built SynthDoG (Synthetic Document Generator). Each generated sample composes four components: a background sampled from ImageNet, a paper texture sampled from collected photographs, text sampled from Wikipedia, and a layout produced by a rule-based algorithm that randomly stacks grids, with image rendering techniques from earlier synthetic text generation work applied to mimic real documents ^[2]. Using SynthDoG with English, Chinese, Japanese, and Korean Wikipedia, the authors generated 0.5 million samples per language, 2 million synthetic images in total ^[2].

The released donut-base model was pretrained for 200,000 steps on 64 NVIDIA A100 GPUs with a mini-batch size of 196, using the Adam optimizer with a scheduled learning rate whose initial value was selected from the range 1e-5 to 1e-4, which the GitHub repository reports as roughly 2.5 days of training ^[2][3]. An earlier prototype checkpoint, donut-proto, was trained at 2048x1536 resolution on 8 V100 GPUs using SynthDoG data only ^[3].

Benchmarks

Donut was evaluated by fine-tuning on three task families across six datasets: document classification (RVL-CDIP, 400K images in 16 classes), document information extraction (the public CORD receipt benchmark and Chinese train-ticket benchmark, plus two private NAVER production datasets of Japanese business cards and Korean receipts), and document visual question answering (DocVQA, 50K questions over more than 12K documents, scored by ANLS, an edit-distance-based metric) ^[2]. Information extraction is scored by field-level F1 and by a tree-edit-distance (TED) based accuracy that credits partially correct nested structures ^[2]. Numbers below are from the ECCV 2022 camera-ready tables; inference times were measured on an NVIDIA P40 GPU, and the parameter counts for OCR-based baselines exclude the OCR engine itself ^[2].

Scroll sideways for more →

On RVL-CDIP, Donut edged out LayoutLMv2 with 143M parameters against 200M plus an OCR engine, at roughly twice the speed ^[2]. On all four information-extraction datasets it posted the best F1 and TED accuracy among the compared systems, which included BERT-, BROS-, and LayoutLMv2-based pipelines and the OCR-dependent IE models SPADE and WYVERN ^[2]. On DocVQA, Donut's 67.5 ANLS was competitive with same-scale OCR pipelines but well below the 86.7 of a DocVQA-specialized LayoutLMv2-Large model trained with extra data; on the handwritten-document subset, however, Donut's 72.1 beat that model's 67.3, since handwriting is exactly where OCR errors cap the pipeline's ceiling ^[2]. The paper also highlights a low-data result: on CORD, Donut surpassed LayoutLMv2's accuracy using only 80 training samples, a tenth of the dataset ^[2].

Release and ecosystem

NAVER released the full stack on GitHub (clovaai/donut) under the MIT license: training and inference code, the SynthDoG generator, the 0.5M-sample synthdog-en/zh/ja/ko datasets, the donut-base and donut-proto pretrained checkpoints, and task-specific fine-tunes ^[3]. The fine-tuned checkpoints and their repository-reported scores are donut-base-finetuned-cord-v2 (91.3 on CORD), donut-base-finetuned-rvlcdip (95.3), donut-base-finetuned-docvqa (67.5), and donut-base-finetuned-zhtrainticket (98.7) ^[3].

Donut was added to the Hugging Face Transformers library in version 4.22.0 (September 14, 2022), contributed by Niels Rogge ^[6]. It is implemented as a VisionEncoderDecoderModel pairing a dedicated DonutSwin encoder with a BART decoder, with a DonutProcessor wrapping the image processor and tokenizer; the library documentation's document-question-answering pipeline examples run on donut-base-finetuned-docvqa ^[5]. The model remains in active use: as of July 2026 the donut-base repository showed roughly 72,000 monthly downloads, 254 likes, 487 fine-tuned derivatives, and 42 community Spaces ^[4]. The model card describes donut-base as a base model meant to be fine-tuned on downstream tasks such as document image classification or document parsing ^[4].

Influence and limitations

Donut demonstrated that a single pixels-to-text Transformer could match or beat OCR pipelines on real document workloads, and its recipe (Swin encoder, shallow multilingual BART decoder, read-the-text pretraining, prompt-selected tasks, JSON-as-token-sequence output) became a reference design for OCR-free document AI. The most direct descendant is Nougat, Meta AI's 2023 model for converting scientific PDFs into markup, whose paper states plainly "We build on the Donut architecture," pairing a Swin encoder with an mBART decoder following Kim et al. ^[7]. The broader idea of end-to-end document reading without a separate OCR stage has since been pursued in vision-language models and in later OCR-focused systems such as DeepSeek-OCR and olmOCR, which likewise decode text directly from page images.

The paper is candid about the main limitation: performance depends heavily on input resolution. Accuracy on DocVQA grows rapidly with larger inputs because document scans contain many tiny texts, but bigger images make the vanilla attention in the encoder much more expensive; the authors note that efficient attention mechanisms could help but chose to keep the architecture simple ^[2]. Qualitative examples show Donut missing small print in large pages that OCR-based systems catch ^[2]. The gap to specialized OCR pipelines on text-dense VQA, and the cost of scaling resolution, are exactly the axes later OCR-free models worked on.

See also

• OCR models
• Nougat
• LayoutLM
• Swin Transformer
• Document question answering models
• Vision-language model

References

1. ^Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S. "OCR-free Document Understanding Transformer." arXiv:2111.15664, v1 November 30, 2021, v5 October 6, 2022. arxiv.org/...2111.15664
2. ^Kim, G. et al. "OCR-free Document Understanding Transformer." Proceedings of ECCV 2022. ecva.net/...136880493.pdf
3. ^clovaai/donut GitHub repository (official code, models, and SynthDoG). github.com/...donut
4. ^naver-clova-ix/donut-base model card, Hugging Face. huggingface.co/...donut-base
5. ^"Donut," Hugging Face Transformers documentation. huggingface.co/...donut
6. ^Hugging Face Transformers release v4.22.0 (September 14, 2022). github.com/...v4.22.0
7. ^Blecher, L., Cucurull, G., Scialom, T., Stojnic, R. "Nougat: Neural Optical Understanding for Academic Documents." arXiv:2308.13418, August 25, 2023. arxiv.org/...2308.13418