Open Images

Open Images is a large annotated image dataset published by Google for computer vision research. It contains roughly 9 million photographs collected from Flickr, annotated with about 61.4 million human-verified image-level labels spanning 20,638 classes, 15.85 million bounding boxes across 600 object classes, 2.79 million instance segmentation masks, 3.28 million visual relationship annotations, 675,155 localized narratives, and 66.4 million point-level labels ^[1][2]. Google released the first version in September 2016 and grew the annotations through seven major releases, the most recent being Open Images V7 in October 2022 ^[3][10].

The dataset was designed to push image understanding past single-label classification. Its images show complex scenes, averaging 8.3 bounding boxes per image on the box-annotated subset, and every annotation layer (labels, boxes, masks, relationships, narratives, points) applies to the same underlying pool of photographs, so the different tasks can be studied together ^[6][10]. Open Images also anchored two public competitions, the Open Images Challenges of 2018 and 2019, run on Kaggle with prize pools of 50,000 and 75,000 US dollars ^[12][13].

Licensing works on two levels: Google licenses the annotations themselves under CC BY 4.0, while the photographs carry CC BY 2.0 licenses from their original Flickr uploaders. Google explicitly warns that it "make[s] no representations or warranties regarding the license status of each image" and tells users to verify licenses themselves ^[4][5].

Origins

Google announced Open Images on September 30, 2016, in a blog post by engineers Ivan Krasin and Tom Duerig, crediting a collaboration between Google, Carnegie Mellon University, and Cornell University ^[3]. The initial release was a list of about 9 million image URLs with labels across more than 6,000 categories. The labels were populated automatically by a vision model similar to the Google Cloud Vision API, with human raters verifying the machine's output on the validation set to remove false positives; each image carried around 8 labels on average ^[3].

The announcement positioned the dataset against ImageNet, arguing that its label space covers "more real-life entities than the 1000 ImageNet classes" used in the ILSVRC competition, and that the dataset is large enough to train a deep learning model from scratch. As a demonstration, the team trained an Inception v3 model on Open Images annotations and released it for fine-tuning and style-transfer experiments ^[3].

The data splits have been stable since 2017: 9,011,219 training images, 41,620 validation images, and 125,436 test images. The original V1 validation set was partitioned into the current validation and test sets in V2 to make evaluation more tractable ^[2][4].

Version history

Scroll sideways for more →

The V4 release marked the dataset's shift toward object detection at scale, with boxes drawn by professional annotators on 1.9 million images ^[6]. It is also the version described in the reference paper by Alina Kuznetsova and colleagues, published in the International Journal of Computer Vision in 2020, which reports 9.2 million images, 15.4 million boxes, and 30.1 million image-level labels for 19.8 thousand concepts, and notes that the images were "collected from Flickr without a predefined list of class names or tags, leading to natural class statistics" ^[7].

Annotation types

Image-level labels

V7 carries 61.4 million image-level labels covering 20,638 classes, of which 9,668 are considered trainable ^[1][2]. On the training split these break down into 21.1 million human-verified positive labels and 37.6 million human-verified negatives; negatives record that a class was checked and is absent, which matters for training classifiers on incomplete label sets ^[2]. A further 164.8 million machine-generated labels ship for the training split. Google's own V3 documentation cautioned that the machine-generated set has a "substantial false positive rate," which is why the human-verified subset is treated as the reliable core ^[2][5].

Bounding boxes

The dataset provides 15,851,536 bounding boxes for 600 object classes on 1.9 million images: 14,610,229 boxes on the training split, 303,980 on validation, and 937,327 on test ^[1][2]. The 600 boxable classes are organized in a class hierarchy distributed with the metadata ^[15]. These 1.9 million box-annotated images form the dataset's densely annotated subset, and later annotation layers concentrate on them ^[10][15].

Segmentation masks

Added in V5, the masks cover 2,785,498 object instances across 350 of the 600 boxable classes: 2,686,666 masks on 944,037 training images, plus 24,730 validation and 74,102 test masks ^[2]. Training masks were produced with an interactive process in which human annotators iteratively corrected the output of a segmentation model, reaching 84 percent intersection-over-union agreement with fully manual drawing; the validation and test masks (about 99k) were drawn entirely by hand for benchmark quality ^[8]. This made Open Images one of the largest instance segmentation resources available at the time.

Visual relationships

Open Images V7 includes 3,284,280 relationship annotations drawn from 1,466 distinct triplets of the form subject-relation-object, such as "woman playing guitar" ^[1][6]. The task was introduced in V4 with 375k annotations over 57 classes, then expanded roughly tenfold in V6, which also folded in 2.5 million annotations of standalone human actions like jumping, smiling, or laying down ^[7][9].

Localized narratives

Localized narratives, introduced with V6, are a multimodal annotation format in which an annotator describes an image aloud while moving the mouse over the regions being described. Voice, transcription, and mouse trace are synchronized, so nearly every word of the description is grounded to a region of the image ^[9][11]. Open Images carries 675,155 narratives covering about 671,000 images, up from the 500,000 images covered at the V6 launch ^[1][2][9]. The format was developed by Jordi Pont-Tuset and colleagues at Google Research and published at ECCV 2020; beyond Open Images, the same team annotated the whole of COCO, Flickr30k, and ADE20K, for a combined 849k images ^[11]. Google noted at launch that the mouse traces collected totaled roughly 6,400 km in length and that the narratives would take about a year and a half to listen to end to end ^[9]. The grounded text makes the corpus useful for image captioning and other vision-language model research ^[11].

Point labels

V7's headline addition was point-level annotation: sparse points inside an image, each tagged yes or no for a specific class. An ML model proposed points and framed yes/no questions (for example, "is this point on a pumpkin?"), and human annotators answered them at an average speed of 1.1 seconds per question, with several annotators' answers aggregated into a final yes, no, or unsure verdict ^[10]. The campaign collected 38.6 million new point annotations (12.4 million of them positive), which Google described as roughly two years of human annotation effort; combined with points derived from existing sources, the dataset totals 66,391,027 point labels over 1.4 million images and 5,827 classes ^[1][2][10]. Because points are cheap, the label space reaches far beyond the 600 boxable categories and covers both "thing" and "stuff" classes, the distinction that panoptic segmentation also builds on ^[10][15]. V7 also shipped new visualizers, including an all-in-one view that overlays labels, boxes, masks, relationships, narratives, and points on the densely annotated subset ^[10].

Open Images Challenges

Google ran public competitions on the dataset in 2018 and 2019, both hosted on Kaggle and both concluding with workshops at major vision conferences.

Scroll sideways for more →

The 2018 edition was announced together with V4 and framed as a step up from earlier benchmark competitions such as PASCAL VOC, ILSVRC, and COCO, with an expanded category set that included classes like "fedora" and "snowman" ^[6][12]. The 2019 edition added the instance segmentation track over 300 classes, built on the new V5 masks ^[8][13].

Licensing and access

All annotations are licensed by Google under CC BY 4.0. The images themselves were sourced from Flickr as CC BY 2.0 content, but the license status of each photograph rests with its uploader: the project's documentation states plainly that "we make no representations or warranties regarding the license status of each image and you should verify the license for each image yourself" ^[4][5]. Per-image metadata preserves the original Flickr landing URL and license for exactly this purpose ^[15].

Google distributes the annotation files directly, while packaged downloads of the images are provided through the Common Visual Data Foundation (CVDF) ^[15]. Since May 2021 the dataset has also been integrated with Voxel51's open-source FiftyOne tool, which handles downloading, visualization, and evaluation ^[16].

Extensions

The Open Images Extended collection gathers companion datasets that add images or annotations beyond the core release ^[14]:

• Crowdsourced extension: more than 382,000 images across 6,000+ categories donated by users of the Google Crowdsource Android app, predominantly from India with contributions from the Middle East, Africa, and Latin America, intended to broaden the dataset's geographic coverage. It ships donated-and-verified positive image-level labels but no bounding boxes ^[14].
• MIAP (More Inclusive Annotations for People): about 100,000 images containing people, re-annotated with nearly 100,000 additional person boxes plus gender-presentation and age-range attributes, published at AIES 2021 to support fairness research on bias in vision systems ^[14].
• HierText: 11,639 Open Images photographs with about 1.2 million words annotated hierarchically (word, line, paragraph) for OCR research ^[14].

Comparison with ImageNet and COCO

The three datasets serve different purposes. ImageNet, organized over the WordNet hierarchy, indexes 14,197,122 images across 21,841 synsets and drove the image-classification era; its annotations are primarily one label per image ^[18]. COCO contains 328,000 images of 91 object categories with 2.5 million labeled instances and pioneered large-scale per-instance segmentation, but at a much smaller image count ^[17]. Open Images sits between and beyond them: ImageNet-scale image counts, but with multi-label annotations, roughly six times as many localized object instances as COCO (about 16 million boxes over 600 classes), plus masks, relationships, narratives, and points on the same images ^[1][2][7]. The V4 paper argues that collecting images without a predefined class list gives Open Images more natural class statistics than datasets built by searching for specific categories ^[7].

Limitations

The annotation layers are deliberately partial. Boxes exist for 1.9 million of the 9 million images, masks for only 350 of the 600 boxable classes, and narratives and points for subsets again, so exhaustive labeling cannot be assumed outside the densely annotated core ^[1][2]. The 164.8 million machine-generated labels carry a substantial false positive rate by Google's own description and need the human-verified subset as ground truth ^[5]. Because images live under individual Flickr licenses, reuse requires per-image license checking, and commercial users cannot rely on the dataset-level CC BY statement alone ^[4]. The project has also documented overlap between Open Images and other Flickr-derived datasets: a June 2022 analysis mapped duplicate images shared with Flickr30k and COCO, which matters for anyone evaluating models trained on one and tested on another ^[16]. Finally, the existence of the crowdsourced extension reflects a known geographic skew in the Flickr source pool, which Google sought to offset with donated images from underrepresented regions ^[14].

Status

Open Images V7 remains the current release. The project's news page lists no updates after the V7 announcement of October 25, 2022, and no successor version had been announced as of mid-2026 ^[16]. The dataset continues to be distributed through the official site, CVDF, and FiftyOne, and its annotations remain a standard resource for data labeling methodology, detection and segmentation benchmarks, and grounded vision-language research ^[11][15][16].

See also

• ImageNet
• COCO Dataset
• PASCAL VOC
• Segment Anything Model and Dataset (SAM and SA-1B)
• LAION-5B
• Data Labeling

References

1. ^Open Images V7 official site, Google. storage.googleapis.com/...web
2. ^Open Images V7: Facts and figures, Google. storage.googleapis.com/...factsfigures_v7
3. ^Krasin, I. and Duerig, T., "Introducing the Open Images Dataset," Google Research Blog, September 30, 2016. research.google/...oducing-the-open-images-dataset
4. ^Open Images dataset README V2 (July 20, 2017), openimages/dataset repository, GitHub. github.com/...READMEV2.md
5. ^Open Images dataset README V3 (November 16, 2017), openimages/dataset repository, GitHub. github.com/...READMEV3.md
6. ^"Announcing Open Images V4 and the ECCV 2018 Open Images Challenge," Google Research Blog, April 30, 2018. research.google/...eccv-2018-open-images-challenge
7. ^Kuznetsova, A. et al., "The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale," International Journal of Computer Vision, 2020. arxiv.org/...1811.00982
8. ^"Announcing Open Images V5 and the ICCV 2019 Open Images Challenge," Google Research Blog, May 8, 2019. research.google/...iccv-2019-open-images-challenge
9. ^Pont-Tuset, J., "Open Images V6, Now Featuring Localized Narratives," Google Research Blog, February 26, 2020. research.google/...-featuring-localized-narratives
10. ^Benenson, R., "Open Images V7, Now Featuring Point Labels," Google Research Blog, October 25, 2022. research.google/...s-v7-now-featuring-point-labels
11. ^Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., and Ferrari, V., "Connecting Vision and Language with Localized Narratives," ECCV 2020. arxiv.org/...1912.03098
12. ^Open Images Challenge 2018 official page, Google. storage.googleapis.com/...challenge
13. ^Open Images Challenge 2019 official page, Google. storage.googleapis.com/...challenge2019
14. ^Open Images Extended, Google. storage.googleapis.com/...extended
15. ^Open Images V7: Download, Google. storage.googleapis.com/...download_v7
16. ^Open Images: News, Google. storage.googleapis.com/...news
17. ^Lin, T.-Y. et al., "Microsoft COCO: Common Objects in Context," ECCV 2014. arxiv.org/...1405.0312
18. ^About ImageNet, image-net.org. image-net.org/about