# Object detection

> Source: https://aiwiki.ai/wiki/object_detection
> Updated: 2026-07-29
> Fact-checked: 2026-07-29
> Categories: Artificial Intelligence, Computer Vision, Deep Learning, Machine Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Object detection." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/object_detection
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Object detection** is a [computer vision](https://aiwiki.ai/wiki/computer_vision) task that finds instances of interest in an image and assigns each one a category. In the standard two-dimensional formulation, a detector returns a set of axis-aligned [bounding boxes](https://aiwiki.ai/wiki/bounding_box), class labels, and scores. This is a set-prediction problem because an image can contain no target objects, one object, or many objects in different locations.[1] Detection is used both as a research task in its own right and as a component of systems that need localized visual information.

Object detection differs from [image classification](https://aiwiki.ai/wiki/image_classification_models), which normally assigns labels to an image as a whole, and from [image segmentation](https://aiwiki.ai/wiki/image_segmentation), which assigns labels or instance masks at pixel level. It is also distinct from phrase [grounding](https://aiwiki.ai/wiki/grounding), although open-vocabulary detectors increasingly connect the two tasks. A conventional closed-set detector is trained and evaluated on a fixed category vocabulary. An open-vocabulary detector accepts text or image queries and attempts to localize categories that need not be among a small fixed set.

A detection score is not automatically the probability that a prediction is correct. It can combine class evidence, objectness, and sometimes localization quality, and it can be miscalibrated. Thresholds therefore need validation for the model, data distribution, and operational cost of false positives and false negatives.[3]

This article focuses on two-dimensional object detection in still images. Three-dimensional detection, rotated-box detection, video tracking, and instance segmentation use related ideas but have different outputs and evaluation rules.

## Task definition

### Output representation

For an input image `I`, a detector can be represented as a function:

```
f_theta(I) = {(b_i, c_i, s_i) | i = 1, ..., N}
```

Here, `b_i` is a predicted box, `c_i` is a category label, `s_i` is a score, and `N` is the number of retained predictions. A box is commonly encoded by two corners `(x_1, y_1, x_2, y_2)`, or by its center, width, and height. Coordinates may be expressed in pixels or normalized by image dimensions. The choice must be recorded because resizing, padding, and coordinate conversion can otherwise shift the final boxes.

Ground-truth annotations define which visible regions count as objects, which category names are valid, and how difficult cases are handled. These are properties of a dataset and annotation policy, not universal facts about an image. For example, one dataset may label a partially occluded object while another may omit it, and a category may be split into several fine-grained classes in one taxonomy but combined in another.

Closed-set detectors usually include an explicit or implicit background outcome. At deployment, an object outside the training vocabulary may be assigned to a known category, receive a low score, or be ignored. Open-vocabulary methods instead compare region features with language or image-query representations, but they still depend on their pretraining data, prompt wording, localization head, and decision thresholds.

### Relation to neighboring tasks

Several visual tasks use similar backbones but answer different questions:

- **Image classification** asks which categories describe an entire image. It does not require one location per object.
- **Object localization** often assumes one primary object and predicts its location. Detection allows multiple instances and categories.
- **Semantic segmentation** assigns a class to each pixel but does not necessarily distinguish two objects of the same class.
- **Instance segmentation** predicts a separate pixel mask for each object instance. Many systems extend a detector with a mask head.
- **Keypoint estimation** predicts landmarks such as joints or corners. Keypoints can also serve as an internal representation for a detector.
- **Visual grounding** maps a word, phrase, or referring expression to one or more image regions.
- **Tracking** associates detections across frames. Detection errors and identity association errors are evaluated separately.

The distinction matters when reading benchmark results. A box AP result cannot be substituted for mask AP, and a zero-shot grounding result does not show that a detector has learned a stable closed-set taxonomy.

## Detection pipeline

Modern detectors are trained with [machine learning](https://aiwiki.ai/wiki/machine_learning), usually [deep learning](https://aiwiki.ai/wiki/deep_learning), but their pipelines still contain several separable choices. Two systems with the same model-family name may differ in image resolution, backbone, feature pyramid, label assignment, data augmentation, pretraining, loss functions, and postprocessing.

### Feature extraction

A feature extractor transforms the image into one or more spatial feature maps. Convolutional backbones preserve local spatial structure while increasing receptive field and semantic abstraction. Transformer and hybrid backbones divide an image into patches or tokens and use attention or state-space operations to exchange information. In either case, the detector needs features that retain enough spatial resolution to localize objects.

Object scale creates a basic difficulty. A small object may occupy only a few cells on a deeply downsampled feature map, while a large object needs a wide receptive field. Feature Pyramid Networks introduced a top-down path and lateral connections that combine high-level semantics with higher-resolution maps at several scales, at relatively small additional cost.[6] Variants of multiscale feature fusion are now used in region-based, dense, and transformer detectors.

The backbone is often initialized through [pretraining](https://aiwiki.ai/wiki/pre-training), followed by detector training or fine-tuning. ImageNet classification pretraining was common in early deep detectors. Later systems also use detection pretraining, self-supervised visual pretraining, or vision-language pretraining. A benchmark entry that uses extra data is not directly comparable with one trained only on the target dataset.

### Candidate representations

Detector families differ in how they represent possible objects:

- **Region proposals** are a sparse set of candidate regions. A second stage extracts region features, classifies the proposals, and refines their boxes.
- **Anchors or default boxes** tile one or more feature maps with predetermined scales and aspect ratios. The network classifies and adjusts these candidates.
- **Points and keypoints** represent an object by its center, corners, or eligible foreground locations, with distances or dimensions used to recover the box.
- **Object queries** are learned or image-conditioned vectors decoded into a fixed-size set of predictions. Set matching assigns some queries to ground-truth objects and the rest to a no-object class.

These representations are design patterns, not complete model definitions. An anchor-free detector can still use non-maximum suppression, and a transformer detector can use multiscale convolutional features. "One-stage" also does not guarantee lower end-to-end latency than "two-stage." Runtime depends on the full graph, number of candidates, image size, hardware, software stack, precision, and postprocessing.

### Label assignment and matching

Training requires a rule that pairs predictions or candidate locations with ground-truth objects. Anchor-based systems may use Intersection over Union thresholds to designate positive, negative, and ignored anchors. Point-based systems can assign locations inside a box, sometimes restricting positives to a central region or a scale range. Set-prediction systems such as DETR use bipartite matching, commonly called Hungarian matching, to choose a one-to-one assignment between ground truth and a subset of queries.[1]

The assignment rule affects which examples contribute to the loss. Dense detectors create many more background candidates than positive candidates. Focal loss addresses this imbalance by reducing the contribution of well-classified examples:

```
FL(p_t) = -alpha_t (1 - p_t)^gamma log(p_t)
```

When `gamma` is positive, easy examples receive less weight, allowing harder examples to have greater influence. RetinaNet used this loss with a dense, anchor-based detector and a feature pyramid.[5]

### Box and classification losses

A detector generally optimizes at least a classification term and a localization term. Classification can use cross-entropy, focal loss, or a related objective. Localization can use coordinate losses such as `L1`, overlap-based losses, or a distribution over possible offsets.

For two regions `A` and `B`, Intersection over Union is:

```
IoU(A, B) = |A intersect B| / |A union B|
```

IoU ranges from zero for non-overlapping regions to one for identical regions. It is invariant to a common scale factor, but ordinary IoU has no overlap gradient when boxes do not intersect. Generalized IoU adds a penalty based on the smallest enclosing region, making an overlap-derived loss useful even for non-overlapping boxes.[4] Later losses modify this idea with center-distance or aspect-ratio terms. The exact loss named in a paper should not be inferred from the model-family name.

Multi-task loss weights determine the balance among classification, box regression, objectness, centerness, and auxiliary decoder losses. They can change optimization even when the inference architecture remains the same. Some transformer detectors apply an auxiliary loss after each decoder layer, while other systems supervise only the final output.

### Postprocessing

Dense detectors often produce many overlapping predictions for the same object. Non-maximum suppression sorts boxes by score and removes lower-scoring boxes whose overlap with a selected box exceeds a threshold. Class-aware NMS processes categories separately; class-agnostic NMS can suppress boxes across categories. Soft-NMS and box-fusion methods modify rather than simply discard overlapping predictions.

NMS is not a universal requirement. DETR was formulated as direct set prediction with a one-to-one matching loss, and its standard inference procedure does not require anchors or NMS.[1] Other transformer detectors may still use proposal selection, duplicate filtering, or task-specific postprocessing. A claim that "object detectors use NMS" is therefore too broad.

Thresholding and top-k limits are also part of the evaluated system. Changing them can alter precision, recall, latency, and memory. Deployment evaluation should include decoding, resize and normalization, data transfer, NMS or other filtering, and output conversion rather than timing only the neural-network forward pass.

## Historical development

### Hand-engineered features and sliding windows

Early practical detectors scanned a classifier across position and scale. The 2001 Viola-Jones face detector combined three influential components: the integral image for rapidly computing Haar-like features, AdaBoost for selecting and combining features, and a cascade that rejected most background windows early. The reported system processed 384 by 288 pixel images at 15 frames per second on a 700 MHz Pentium III.[7] That speed is a historical paper result, not a comparison with modern hardware.

Dalal and Triggs introduced Histograms of Oriented Gradients for pedestrian detection. Their pipeline computed fine-scale image gradients, accumulated orientation histograms in local cells, normalized overlapping blocks, and trained a linear [support vector machine](https://aiwiki.ai/wiki/support_vector_machine_svm).[8] The representation was designed to capture local shape while reducing sensitivity to illumination and contrast.

Deformable Part Models represented a category with a root template and movable part templates across a feature pyramid. Mixture components handled substantial appearance variation, and latent-SVM training alternated between latent configuration selection and discriminative optimization with mined hard negatives.[9] These systems established ideas such as multiscale search, hard-negative mining, and structured object representation that remained relevant after learned image features became dominant.

### Region-based convolutional detectors

R-CNN applied a [convolutional neural network](https://aiwiki.ai/wiki/convolutional_neural_network) separately to roughly 2,000 region proposals per image, converted each region into a fixed-size input, and classified the resulting [feature vector](https://aiwiki.ai/wiki/feature_vector) with class-specific SVMs. The paper reported 53.3 percent mean average precision on PASCAL VOC 2012, more than a 30 percent relative improvement over the prior best result.[10] Its multistage training and repeated computation per proposal were expensive.

Fast R-CNN shared convolutional computation over the whole image. A region-of-interest pooling layer extracted a fixed-size feature for each proposal, and a single network jointly predicted category scores and refined coordinates. With VGG16, the paper reported training nine times faster and testing 213 times faster than its R-CNN implementation, while noting that its timing excluded proposal generation.[11]

[Faster R-CNN](https://aiwiki.ai/wiki/faster_r_cnn) replaced external proposal generation with a Region Proposal Network that shared full-image convolutional features with the detector. The VGG16 system reported 5 frames per second including all steps, with 73.2 percent mAP on VOC 2007 and 70.4 percent on VOC 2012 using 300 proposals per image.[12] These values belong to the paper's hardware, datasets, and evaluation protocol.

Feature Pyramid Networks then supplied semantically strong features at several resolutions through a top-down and lateral architecture.[6] The combination of proposal networks, multiscale features, region pooling, and joint classification and regression became a durable two-stage template. "Two-stage" describes the proposal and refinement structure; it does not imply that the feature extractor is executed twice.

### Dense one-stage and point-based detectors

[YOLO](https://aiwiki.ai/wiki/yolo) reframed detection as a single-network regression problem over the full image. The original paper reported 45 frames per second for its base model and 155 for Fast YOLO, with the base system obtaining 63.4 percent mAP on VOC 2007. It also reported more localization errors than some competing systems.[13] Later models called YOLO were produced by different authors and organizations, so the name does not identify one continuous architecture, license, or benchmark protocol.

[SSD](https://aiwiki.ai/wiki/ssd_object_detection) predicted class scores and box offsets for default boxes on feature maps of multiple resolutions. In the final paper version, SSD300 reported 74.3 percent mAP on VOC 2007 at 59 frames per second on an NVIDIA Titan X, and SSD512 reported 76.9 percent mAP.[14] The input size, GPU, data augmentation, and paper version are necessary context for those numbers.

RetinaNet paired a feature pyramid with dense anchors and focal loss. Its best ResNet-101-FPN model reported 39.1 COCO test-dev AP at 5 frames per second.[5] The work showed that foreground-background imbalance, rather than an unavoidable architectural ceiling, was a major reason dense one-stage training had lagged.

CornerNet represented a box by paired top-left and bottom-right keypoints, used corner pooling to help localize the keypoints, and grouped corresponding corners. Its 42.1 COCO AP result used multiscale testing.[15] CenterNet's "Objects as Points" formulation instead represented each object by the center point of its box and regressed size and offset at that center.[16] It should not be conflated with other projects that also used the CenterNet name.

FCOS made a prediction at foreground feature-map locations and regressed distances to the left, top, right, and bottom sides of a box. A centerness branch reduced low-quality boxes far from the object center. The method was anchor-free and proposal-free but still used NMS. Its improved ResNeXt-64x4d-101-FPN configuration reported 44.7 COCO AP with single-model, single-scale testing.[17]

Anchor-based and anchor-free methods both remain useful. Anchors encode prior scales and aspect ratios but introduce assignment rules and hyperparameters. Point-based methods simplify this representation but still require choices about eligible locations, scale ranges, and ambiguous overlaps. Accuracy and speed must be established for an implementation and workload rather than inferred from the representation alone.

### Set prediction and detection transformers

[DETR](https://aiwiki.ai/wiki/detr) combined a convolutional backbone with a [Transformer](https://aiwiki.ai/wiki/transformers) encoder-decoder, learned object queries, and a one-to-one bipartite matching loss. Its ResNet-50 model reported 42.0 COCO validation AP after a 500-epoch schedule, with stronger large-object AP but weaker small-object AP than a comparably reported Faster R-CNN baseline.[1] DETR simplified several hand-designed components, but the original version converged slowly and had limited high-resolution multiscale processing.

Deformable DETR made attention sample a small set of points around reference locations on multiscale feature maps. The paper reported 43.8 COCO validation AP for its basic ResNet-50 model after 50 epochs, compared with 42.0 AP for the original DETR after 500 epochs in the same table. Iterative refinement and a two-stage variant raised the reported validation results further.[18] A 46.9 AP test-dev result in that paper refers to a particular ResNet-50 configuration and split, not the basic 43.8 AP validation row.

DINO improved DETR-style training with contrastive denoising, mixed query selection, and a look-forward-twice box update. With a Swin-L backbone pretrained on ImageNet-22K and detector pretraining on Objects365, the paper reported 63.2 COCO validation AP and 63.3 test-dev AP without test-time augmentation.[19] These scores therefore should not be compared as if the model used only COCO training or a ResNet-50 backbone.

RT-DETR optimized a DETR-style detector for measured end-to-end latency with an efficient hybrid encoder, uncertainty-minimal query selection, and a decoder whose layer count could be adjusted after training. The paper reported 53.1 and 54.3 COCO AP for its ResNet-50 and ResNet-101 models at 108 and 74 frames per second on an NVIDIA T4 using TensorRT FP16. Objects365 pretraining raised the reported AP to 55.3 and 56.2.[20]

D-FINE refined box coordinates as probability distributions and used a localization self-distillation method across decoder layers. The peer-reviewed ICLR 2025 paper reported 54.0 and 55.8 COCO AP for its L and X models at 124 and 78 frames per second, measured as end-to-end TensorRT FP16 latency on an NVIDIA T4. With Objects365 pretraining, the reported values were 57.1 and 59.3 AP.[21]

RF-DETR, accepted at ICLR 2026, combined an internet-scale pretrained visual backbone, a lightweight detection transformer, and weight-sharing neural architecture search for task-specific accuracy-latency tradeoffs. Its paper reported that the 2XL variant was the first real-time detector to exceed 60 COCO AP, explicitly qualifying this as an author "to the best of our knowledge" claim.[22] The result does not establish that every RF-DETR checkpoint is faster or more accurate than every alternative, and it does not make results from different latency setups directly comparable.

The progression from convolutional detectors to transformer detectors is not a clean replacement. Current systems mix convolutional pyramids, pretrained visual encoders, attention, query selection, denoising objectives, NMS-free set prediction, and hardware-specific graph optimization. Architecture labels are useful for describing a design, but they are not sufficient evidence for deployment performance.

## Open-vocabulary detection

Closed-set benchmarks assume a predetermined vocabulary. Open-vocabulary detection instead uses language or image queries to define categories at inference, with the goal of transferring beyond a short fixed label list. This connects detection with vision-language representation learning and [natural language processing](https://aiwiki.ai/wiki/natural_language_processing).

GLIP reformulated closed-set object detection as phrase grounding and pretrained on 27 million grounding examples, including 3 million human-annotated examples and 24 million web-derived image-text pairs. The paper reported 49.8 zero-shot COCO AP without using COCO images during pretraining, and 60.8 validation AP after COCO fine-tuning.[23] These results use different settings and should not be placed in one unqualified ranking.

OWL-ViT adapted a contrastively pretrained [Vision Transformer](https://aiwiki.ai/wiki/vision_transformer) with minimal changes for text-conditioned and image-conditioned detection. Its design showed how [CLIP](https://aiwiki.ai/wiki/clip)-style image-text pretraining can transfer to localization, while the detector still required objectness and box prediction heads trained with localized data.[24]

Grounding DINO combined a DINO-style detector with grounded pretraining, a cross-modal feature enhancer, language-guided query selection, and a cross-modality decoder. The ECCV 2024 paper reported 52.5 zero-shot COCO AP under its definition of zero-shot, meaning that the COCO training split was not used, and 26.1 mean AP on the ODinW benchmark.[25] "Zero-shot" therefore describes the target split policy, not absence of all related concepts from pretraining.

Grounding DINO 1.5 was released as a 2024 technical preprint rather than a peer-reviewed paper. It described training on more than 20 million grounded images and reported 54.3 zero-shot COCO AP for its Pro model. Its Edge result of 75.2 frames per second was paired with TensorRT optimization and a separate 36.2 AP result on LVIS minival.[26] Those figures should remain labeled as preprint-reported results with their respective datasets and runtime conditions.

Open-vocabulary systems introduce additional evaluation questions. Prompt spelling, synonyms, category descriptions, negative prompts, and text-encoding choices can change outputs. A model may localize a broad concept while failing on fine-grained categories, attributes, or counts. Its vocabulary is not literally unlimited because behavior is bounded by pretraining, representations, and the inference interface.

## Evaluation

### Matching predictions to ground truth

Evaluation first decides whether a prediction corresponds to a ground-truth object. A common procedure sorts predictions by score and greedily matches each one to an unmatched ground-truth box of the same class when IoU exceeds a threshold. A matched prediction is a true positive, an unmatched prediction is a false positive, and an unmatched ground-truth object is a false negative. Rules for ignored regions, crowd annotations, maximum detections, and category handling can change the counts.

[Precision](https://aiwiki.ai/wiki/precision) and [recall](https://aiwiki.ai/wiki/recall) are:

```
Precision = TP / (TP + FP)    Recall = TP / (TP + FN)
```

Raising a score threshold usually removes both true and false detections, producing a precision-recall tradeoff. A single thresholded precision value does not summarize ranking quality over all thresholds.

### Average precision and average recall

[Average precision](https://aiwiki.ai/wiki/average_precision) summarizes the precision-recall curve for one category and matching criterion. Implementations differ. PASCAL VOC 2007 used an 11-point interpolated definition, while later VOC evaluation used all distinct recall changes.[28] COCO uses its own evaluator and should not be described merely as an unspecified area under a curve.

The official COCO evaluator samples 101 recall thresholds from 0.00 through 1.00 and reports AP averaged over 10 IoU thresholds from 0.50 through 0.95 in increments of 0.05. It also reports AP at IoU 0.50 and 0.75, and by small, medium, and large object area. Standard detection summaries cap detections at 100 per image; average-recall summaries also use maximum-detection settings of 1, 10, or 100.[27]

The headline COCO "AP" is thus stricter than AP50. It rewards both classification and localization across several overlap thresholds. AP for small objects can be much lower than AP for large objects because fewer image pixels represent the object and annotation uncertainty occupies a larger fraction of its area.

Average recall measures how many annotated objects are recovered under a specified proposal or detection limit, averaged over overlap thresholds. It is useful when the next pipeline stage can tolerate extra candidates, but its maximum-detection limit must be stated.

### Calibration and operating points

AP evaluates ranking and localization, not whether a score has probabilistic meaning. Two detectors can have similar AP but different calibration. Detection calibration is more complex than classification calibration because an output has a class, a score, and a location, and duplicate predictions interact with matching. Research has documented overconfidence and evaluated calibration errors for both in-domain and shifted detection data.[3]

Operational thresholds should be chosen on representative validation data using application-specific error costs. A threshold chosen to maximize an F1 score on one benchmark may not be appropriate when missed objects and false alarms have unequal consequences. Per-class thresholds can help when base rates and error costs differ, but they add parameters that must be monitored after distribution change.

### Comparability of reported results

Detector scores and speed figures are comparable only when the relevant protocol is aligned. Important variables include:

- dataset version, split, category mapping, and evaluation code;
- training data, [ImageNet](https://aiwiki.ai/wiki/imagenet) or detection pretraining, and synthetic data;
- backbone, parameter count, image resolution, and single-scale or multiscale testing;
- test-time augmentation, ensembling, score thresholds, NMS, and maximum detections;
- latency versus throughput, batch size, warm-up, data transfer, and whether preprocessing and postprocessing are included;
- hardware model, numerical precision, compiler or inference engine, and software versions.

Frames per second from a Titan X, T4, or another accelerator cannot be compared as if the measurements came from one controlled experiment. A paper-reported accuracy-latency point is evidence for that setup, not a hardware-independent property of the model name.

## Datasets

### PASCAL VOC and COCO

[PASCAL VOC](https://aiwiki.ai/wiki/pascal_voc) established annual challenges, standardized annotations, released evaluation software, and used 20 object categories in its later detection tasks. The VOC 2007 dataset contained 9,963 annotated images and 24,640 annotated objects across classification and detection data.[28] Different VOC years and training combinations such as "07+12" must be identified when reporting results.

The [COCO dataset](https://aiwiki.ai/wiki/coco_dataset) was designed around common objects in complex natural scenes. Its original paper described 328,000 images and 2.5 million labeled instances across 91 object types, while the current project overview describes more than 200,000 labeled images, 1.5 million object instances, and 80 object categories for the widely used detection task.[2][38] These figures refer to different scopes and releases; combining them into a single dataset row creates a false count.

COCO annotations include instance masks as well as boxes, allowing the same images to support detection and instance-segmentation evaluation. The 80-category detection taxonomy is broad but still closed-set and concentrated on common objects. Strong COCO AP does not by itself establish performance on a specialized vocabulary or a different imaging domain.

### Larger and long-tailed datasets

[Open Images](https://aiwiki.ai/wiki/open_images) V4 contains 9.2 million images, 15.4 million boxes for 600 object classes, and 375,000 visual-relationship annotations involving 57 classes. It also contains image-level labels, so totals for labels, boxes, and relationships should not be interchanged.[29] Later Open Images releases exist, but V4 paper statistics should remain labeled V4.

[LVIS](https://aiwiki.ai/wiki/lvis) was created for large-vocabulary, long-tailed instance segmentation. The paper planned 2.2 million high-quality instance masks for more than 1,000 entry-level categories in 164,000 images, and the released dataset has more than 1,200 categories. Its rare, common, and frequent category groupings expose the difficulty of learning classes with few annotated instances.[30][39] LVIS mask annotations can be converted to boxes for detection evaluation, but mask AP and box AP remain separate metrics.

The original Objects365 paper described 365 categories, more than 600,000 training images, and more than 10 million boxes. The current official Objects365 V2 overview instead reports 2 million images and 30 million boxes for the same number of categories.[31][40] These are versioned totals. Model papers that say "Objects365 pretraining" should specify the release when the information is available.

Dataset scale does not remove annotation limitations. Category definitions can overlap, rare cases may be inconsistently labeled, and an image can contain valid but unlabeled objects. Training and test images can also share photographers, locations, or acquisition pipelines in ways that make within-dataset performance easier than deployment transfer.

## Research applications

Object detection is studied in many domains, but a benchmark demonstrates a research task rather than proving a complete deployed system.

The Waymo Open Dataset paper described 1,150 driving scenes of 20 seconds each with synchronized camera and lidar data, 2D and 3D boxes, tracking identifiers, and geographic variation across cities.[32] This supports research in [autonomous driving](https://aiwiki.ai/wiki/autonomous_driving) perception, including camera detection and sensor fusion. A detector evaluated on recorded scenes is not, by that result alone, validated for closed-loop driving or every weather and traffic condition.

SKU-110K was introduced for densely packed retail shelves containing many similar, closely spaced products. The paper emphasized overlapping objects and ambiguity between adjacent boundaries, conditions in which ordinary NMS can merge separate instances or retain boxes spanning several objects.[33] The dataset illustrates that high object density changes both localization and duplicate-suppression behavior.

VinDr-CXR released 18,000 chest radiographs annotated by 17 radiologists. Its 22 local labels use rectangles around abnormalities, with 15,000 training images independently labeled by three radiologists and 3,000 test images labeled by consensus of five radiologists.[34] These annotations support localization research, but a box is not a diagnosis and a benchmark score is not evidence of clinical safety.

The Global Wheat Head Dataset combined field images from several countries and institutions to study wheat-head localization under variation in cultivars, growth stages, imaging conditions, and geography. The 2020 release contained 4,700 images and 193,634 wheat-head boxes; the 2021 extension added 1,722 images and 81,553 boxes from additional regions.[35] Keeping the releases separate prevents double counting.

xView was created for object detection in high-resolution overhead imagery. Its paper reported more than one million annotated objects in 60 classes across more than 1,400 square kilometers of imagery.[36] Small objects, large images, unusual viewing angles, and geographic shift make overhead detection different from ordinary ground-level photography.

Other applications include robotics, wildlife monitoring, industrial inspection, document analysis, and assistive interfaces. For each, a detector's taxonomy, minimum object size, sensor characteristics, acceptable latency, and failure costs need to be defined before a benchmark or model is selected.

## Failure modes and deployment considerations

### Localization and recognition errors

Common error patterns include:

- **Missed small objects.** Downsampling can erase detail, while a small coordinate error causes a large IoU change.
- **Occlusion and truncation.** The visible evidence may not uniquely determine the full extent of an object.
- **Crowding.** Neighboring instances overlap, and NMS can suppress a valid object or retain a box that combines several objects.
- **Class confusion.** Visually similar categories, fine-grained taxonomies, or ambiguous labels can produce plausible but incorrect categories.
- **Duplicate detections.** Several candidates can survive for one object when score and overlap thresholds are poorly matched.
- **Background false positives.** Textures or parts can resemble a learned category outside the context represented in training.
- **Long-tail errors.** Classes with few training examples often have weaker classification and localization.

Error analysis should separate classification, localization, duplicate, and background errors. Aggregating everything into AP can hide a failure that is critical for an application.

### Distribution shift

Changes in geography, weather, camera, resolution, compression, viewpoint, object prevalence, and label policy can all shift the input or target distribution. COUNTS, a CVPR 2025 benchmark, contains 14 natural distribution shifts, more than 222,000 samples, and more than 1.196 million boxes. Its experiments found that limitations persisted under out-of-distribution conditions even for larger models and models with extensive pretraining.[37] This is evidence that in-distribution benchmark gains do not eliminate transfer risk.

Evaluation should therefore include data that represents the intended environment, meaningful stress conditions, and foreseeable unknowns. If a system is updated, the full pipeline should be retested because a change in resizing, category mapping, NMS, or calibration can alter behavior even when the checkpoint is unchanged.

### Runtime and resource constraints

Latency, throughput, memory, power, and model size constrain deployment. Reducing input resolution can increase speed but remove small-object detail. [Quantization](https://aiwiki.ai/wiki/quantization) can reduce compute or memory but needs accuracy testing on the target runtime. [Knowledge distillation](https://aiwiki.ai/wiki/knowledge_distillation) can transfer behavior to a smaller model, but it can also transfer the teacher's biases and omissions.

Batch throughput is not the same as single-image latency. A server can increase throughput by batching while making an individual request wait. Edge devices may face sustained thermal or memory limits not represented by a short benchmark. End-to-end measurement should include image decoding, resize and normalization, device transfer, inference, filtering, coordinate transformation, and result serialization.

### Safety and human factors

A detector should not be treated as an oracle. Confidence thresholds need calibrated interpretation, and low scores are not the only source of error because a model can be confidently wrong. High-impact applications need a defined fallback for missing or uncertain detections, audit logs for model and threshold versions, and monitoring for changes in input and error distributions.

Human review is useful only when reviewers receive enough context, time, and authority to intervene. Displaying dozens of unstable boxes can increase workload rather than reduce it. The interface should distinguish model output from verified information and should not imply that a box establishes causation, identity, diagnosis, or intent.

## Current research directions

Research continues along several connected lines:

- **Open-vocabulary and grounded detection** seeks broader category transfer while making text prompts and evaluation more reproducible.
- **Efficient set prediction** reduces training time and decoder cost while preserving one-to-one outputs.
- **Long-tail learning** addresses rare categories without allowing frequent categories to dominate optimization.
- **Domain generalization and adaptation** evaluate and reduce performance loss under natural distribution change.
- **Calibration and uncertainty** aim to make scores more useful for thresholding and escalation.
- **Data efficiency** uses self-supervision, weak labels, synthetic data, or active selection to reduce bounding-box annotation cost.
- **Dense and small-object detection** improves resolution, tiling, feature fusion, and duplicate handling for crowded or overhead scenes.
- **Multimodal perception** combines images with language, lidar, depth, or temporal evidence while keeping task-specific outputs measurable.

No single benchmark covers all of these goals. Progress is best assessed with explicit data provenance, matched evaluation protocols, ablations, and domain-specific error analysis rather than a universal model ranking.

## See also

- [Image recognition](https://aiwiki.ai/wiki/image_recognition)
- [R-CNN](https://aiwiki.ai/wiki/r_cnn)
- [ResNet](https://aiwiki.ai/wiki/resnet)
- [Post-processing](https://aiwiki.ai/wiki/post-processing)

## References

1. [End-to-End Object Detection with Transformers - European Conference on Computer Vision, 2020](https://arxiv.org/abs/2005.12872)
2. [Microsoft COCO: Common Objects in Context - European Conference on Computer Vision, 2014](https://arxiv.org/abs/1405.0312)
3. [Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection - CVPR 2023](https://openaccess.thecvf.com/content/CVPR2023/html/Munir_Bridging_Precision_and_Confidence_A_Train-Time_Loss_for_Calibrating_Object_CVPR_2023_paper.html)
4. [Generalized Intersection over Union: A Metric and a Loss for Bounding Box Regression - CVPR 2019](https://openaccess.thecvf.com/content_CVPR_2019/html/Rezatofighi_Generalized_Intersection_Over_Union_A_Metric_and_a_Loss_for_CVPR_2019_paper.html)
5. [Focal Loss for Dense Object Detection - ICCV 2017](https://openaccess.thecvf.com/content_iccv_2017/html/Lin_Focal_Loss_for_ICCV_2017_paper.html)
6. [Feature Pyramid Networks for Object Detection - CVPR 2017](https://openaccess.thecvf.com/content_cvpr_2017/html/Lin_Feature_Pyramid_Networks_CVPR_2017_paper.html)
7. [Rapid Object Detection using a Boosted Cascade of Simple Features - CVPR 2001](https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf)
8. [Histograms of Oriented Gradients for Human Detection - CVPR 2005](https://doi.org/10.1109/CVPR.2005.177)
9. [Object Detection with Discriminatively Trained Part-Based Models - IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010](https://doi.org/10.1109/TPAMI.2009.167)
10. [Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation - CVPR 2014](https://openaccess.thecvf.com/content_cvpr_2014/html/Girshick_Rich_Feature_Hierarchies_2014_CVPR_paper.html)
11. [Fast R-CNN - ICCV 2015](https://openaccess.thecvf.com/content_iccv_2015/html/Girshick_Fast_R-CNN_ICCV_2015_paper.html)
12. [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks - NeurIPS 2015](https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html)
13. [You Only Look Once: Unified, Real-Time Object Detection - CVPR 2016](https://openaccess.thecvf.com/content_cvpr_2016/html/Redmon_You_Only_Look_CVPR_2016_paper.html)
14. [SSD: Single Shot MultiBox Detector - European Conference on Computer Vision, 2016](https://arxiv.org/abs/1512.02325)
15. [CornerNet: Detecting Objects as Paired Keypoints - European Conference on Computer Vision, 2018](https://openaccess.thecvf.com/content_ECCV_2018/html/Hei_Law_CornerNet_Detecting_Objects_ECCV_2018_paper.html)
16. [Objects as Points - arXiv, 2019](https://arxiv.org/abs/1904.07850)
17. [FCOS: Fully Convolutional One-Stage Object Detection - ICCV 2019](https://openaccess.thecvf.com/content_ICCV_2019/html/Tian_FCOS_Fully_Convolutional_One-Stage_Object_Detection_ICCV_2019_paper.html)
18. [Deformable DETR: Deformable Transformers for End-to-End Object Detection - ICLR 2021](https://openreview.net/forum?id=gZ9hCDWe6ke)
19. [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection - ICLR 2023](https://openreview.net/forum?id=3mRwyG5one)
20. [DETRs Beat YOLOs on Real-Time Object Detection - CVPR 2024](https://openaccess.thecvf.com/content/CVPR2024/html/Zhao_DETRs_Beat_YOLOs_on_Real-time_Object_Detection_CVPR_2024_paper.html)
21. [D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement - ICLR 2025](https://proceedings.iclr.cc/paper_files/paper/2025/hash/6cf58a87e3097e7d1f9be3e8693a93de-Abstract-Conference.html)
22. [RF-DETR: Neural Architecture Search for Real-Time Detection Transformers - ICLR 2026](https://openreview.net/pdf/6c979474ef7804b04ad00f0b04b7f3311e0a6719.pdf)
23. [Grounded Language-Image Pre-training - CVPR 2022](https://openaccess.thecvf.com/content/CVPR2022/html/Li_Grounded_Language-Image_Pre-Training_CVPR_2022_paper.html)
24. [Simple Open-Vocabulary Object Detection with Vision Transformers - European Conference on Computer Vision, 2022](https://www.ecva.net/papers/eccv_2022/papers_ECCV/html/7529_ECCV_2022_paper.php)
25. [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection - European Conference on Computer Vision, 2024](https://doi.org/10.1007/978-3-031-72970-6_3)
26. [Grounding DINO 1.5: Advance the Edge of Open-Set Object Detection - arXiv technical report, 2024](https://arxiv.org/abs/2405.10300)
27. [COCO evaluation implementation - cocodataset/cocoapi](https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/cocoeval.py)
28. [The PASCAL Visual Object Classes Challenge - International Journal of Computer Vision, 2010](https://doi.org/10.1007/s11263-009-0275-4)
29. [The Open Images Dataset V4 - International Journal of Computer Vision, 2020](https://doi.org/10.1007/s11263-020-01316-z)
30. [LVIS: A Dataset for Large Vocabulary Instance Segmentation - CVPR 2019](https://openaccess.thecvf.com/content_CVPR_2019/html/Gupta_LVIS_A_Dataset_for_Large_Vocabulary_Instance_Segmentation_CVPR_2019_paper.html)
31. [Objects365: A Large-scale, High-quality Dataset for Object Detection - ICCV 2019](https://openaccess.thecvf.com/content_ICCV_2019/html/Shao_Objects365_A_Large-Scale_High-Quality_Dataset_for_Object_Detection_ICCV_2019_paper.html)
32. [Scalability in Perception for Autonomous Driving: Waymo Open Dataset - CVPR 2020](https://openaccess.thecvf.com/content_CVPR_2020/html/Sun_Scalability_in_Perception_for_Autonomous_Driving_Waymo_Open_Dataset_CVPR_2020_paper.html)
33. [Precise Detection in Densely Packed Scenes - CVPR 2019](https://openaccess.thecvf.com/content_CVPR_2019/html/Goldman_Precise_Detection_in_Densely_Packed_Scenes_CVPR_2019_paper.html)
34. [VinDr-CXR: An Open Dataset of Chest X-rays with Radiologists' Annotations - Scientific Data, 2022](https://doi.org/10.1038/s41597-022-01498-w)
35. [Global Wheat Head Dataset 2021: More Diversity to Improve the Benchmarking of Wheat Head Localization Methods - Plant Phenomics, 2021](https://doi.org/10.34133/2021/9846158)
36. [xView: Objects in Context in Overhead Imagery - arXiv, 2018](https://arxiv.org/abs/1802.07856)
37. [COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shifts - CVPR 2025](https://openaccess.thecvf.com/content/CVPR2025/html/Li_COUNTS_Benchmarking_Object_Detectors_and_Multimodal_Large_Language_Models_under_CVPR_2025_paper.html)
38. [COCO dataset overview - COCO Consortium](https://cocodataset.org/#home)
39. [LVIS dataset overview - LVIS Consortium](https://www.lvisdataset.org/)
40. [Objects365 dataset overview - Objects365 Consortium](https://www.objects365.org/overview.html)