R-CNN (Regions with CNN features)

R-CNN (short for Regions with CNN features) is a two-stage object detection method that generates about 2,000 candidate region proposals per image with Selective Search, warps each region and runs a convolutional neural network to extract features, then classifies each region with class-specific linear SVMs and refines its box with bounding-box regression ^[1][3]. Introduced by Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik at UC Berkeley, it was first posted to arXiv as technical report 1311.2524 on 11 November 2013 and presented as an oral at CVPR 2014 under the title Rich feature hierarchies for accurate object detection and semantic segmentation ^[1]. R-CNN was the first deep-learning approach to convincingly beat hand-crafted feature methods on PASCAL VOC, raising mean average precision (mAP) on PASCAL VOC 2012 from the prior best of 40.4% to 53.3%, a more than 30% relative improvement ^[1].

The paper is widely considered the moment when deep learning took over object detection. It launched the "R-CNN family" of two-stage detectors, including SPPnet, Fast R-CNN, Faster R-CNN, Mask R-CNN, and Cascade R-CNN ^{[7][8][9][10][11]}, and has been cited over 30,000 times on Google Scholar ^[20]. Girshick received the 2024 Longuet-Higgins Prize from the IEEE Computer Society for the original R-CNN paper, awarded for fundamental contributions in computer vision that have stood the test of time ^[21]. Although R-CNN itself has been superseded for production use, every modern two-stage detector traces its lineage to this 2014 system.

What is R-CNN?

R-CNN is a region-based object detector: instead of sliding a classifier over every possible window, it scores a small set of class-independent region proposals with a deep CNN. The authors describe the contribution in two parts. Per the abstract, "Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost." ^[1] The name R-CNN stands for "Regions with CNN features," reflecting the pipeline of region proposals feeding CNN feature extraction.

Background and historical context

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton's AlexNet won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), dropping top-5 image-classification error from roughly 26% to 15.3% ^[4]. That result revolutionized image classification overnight, but its impact on related tasks was less obvious. Object detection in particular was still dominated by hand-crafted features. The reigning approach was Pedro Felzenszwalb's Deformable Parts Model (DPM), combining Histogram of Oriented Gradients (HOG) descriptors with a star-structured part-based model trained by latent SVM ^[5]. DPM had won the PASCAL VOC "Lifetime Achievement" prize in 2010, and HOG/DPM detectors had crept up to 33-40% mAP after several years of incremental work ^[5].

The central question in late 2012 and early 2013 was whether the deep features that had transformed classification could also transform detection. OverFeat, by Pierre Sermanet and others at NYU, was an early attempt that ran a CNN as a sliding-window detector. R-CNN took a different route: rather than slide a CNN over every possible window, let an inexpensive bottom-up segmentation algorithm propose a few thousand candidate regions and then score each one with a deep CNN. The proposal algorithm of choice was Selective Search, introduced by Jasper Uijlings, Koen van de Sande, Theo Gevers, and Arnold Smeulders in their 2013 International Journal of Computer Vision paper ^[3]. Selective Search produced about 2,000 category-independent proposals per image with around 99% recall on PASCAL VOC ^[3]. The combination, region proposals plus CNN features, worked shockingly well: the first arXiv version of R-CNN appeared 11 November 2013, and by CVPR 2014 the entire research field was pivoting to deep learning for detection ^[1].

Who created R-CNN?

All four authors of the R-CNN paper were at UC Berkeley at the time the work was done ^[1].

Scroll sideways for more →

Girshick had completed his PhD at the University of Chicago under Pedro Felzenszwalb, working on the very DPM detector that R-CNN would supplant ^[5]. He went on to lead Fast R-CNN at Microsoft Research, then to join Facebook AI Research in 2015 and the Allen Institute for AI in 2023 ^[21]. Donahue's contemporaneous DeCAF paper showed that AlexNet features transferred well to many downstream tasks, providing much of the conceptual basis for R-CNN's transfer-learning step. Malik in particular had been pushing his students toward CNN-based detection in late 2012 and 2013.

How does R-CNN work?

R-CNN is a four-module pipeline plus a non-maximum suppression post-processing step ^[1].

Module 1: Region proposals (Selective Search)

For each input image, Selective Search produces about 2,000 category-independent rectangular proposals ^[1][3]. The algorithm first oversegments the image into many small regions using the graph-based method of Felzenszwalb and Huttenlocher, then iteratively merges adjacent regions that are similar in color, texture, size, and shape ^[3]. Each merged region in the hierarchy contributes a candidate bounding box. R-CNN uses Selective Search in its "fast mode"; it runs on the CPU and takes around 2 seconds per image, which becomes a dominant fixed cost of the system ^[1].

Module 2: CNN feature extraction

Each proposal is cropped and warped to a fixed 227 by 227 pixel input, regardless of its original aspect ratio ^[1]. The paper experimented with several warping strategies and settled on anisotropic warp with 16 pixels of context padding around the original proposal ^[1]. The warped crop is pushed through a CNN that produces a 4,096-dimensional feature vector at the second-to-last fully connected layer (fc7) ^[1]. The original R-CNN used Krizhevsky et al.'s AlexNet architecture: 5 convolutional layers, 2 fully connected layers (fc6 and fc7, each 4,096-d), and a final 1,000-way ImageNet classification layer that is discarded for detection ^[1][4]. The paper also explored Matthew Zeiler and Rob Fergus's improved CNN. In follow-on work using the deeper VGG-16 architecture by Simonyan and Zisserman, the same pipeline reached substantially higher accuracy at a much higher compute cost ^[6].

Module 3: Per-class linear SVMs

For each object category there is a separate binary linear SVM that takes the 4,096-d feature vector and outputs a class confidence score ^[1]. With 20 categories on PASCAL VOC, that is 20 SVMs. The SVMs are trained on features extracted from all proposals in the training set, with hard negative mining ^[1][5].

Module 4: Per-class bounding-box regression and NMS

A per-class linear bounding-box regressor takes the pool5 features of a proposal and predicts four offsets (dx, dy, dw, dh) refining the proposal to better match the ground-truth box ^[1]. Bounding-box regression added a substantial chunk of mAP, lifting AlexNet R-CNN from 50.2% to 53.7% on PASCAL VOC 2010 and from 54.2% to 58.5% on PASCAL VOC 2007 ^[1]. Finally, non-maximum suppression is applied per class: detections are sorted by SVM score and any detection with IoU greater than a threshold (typically 0.3) against a higher-scoring detection of the same class is suppressed ^[1].

How is R-CNN trained?

R-CNN's training is famously a multi-stage pipeline, in contrast to the end-to-end training of its successors ^[1][8].

Stage 1: Supervised pre-training on ImageNet

The CNN is pre-trained on the ImageNet ILSVRC 2012 classification task using image-level labels only (1.2 million training images, 1,000 classes) ^[1][4]. The weights are downloaded from Krizhevsky's published model. The transfer-learning insight, that a CNN trained on a generic large image-classification task could be adapted to detection with much less labeled data, was one of the paper's most influential contributions ^[1]. The paper notes that the "supervised pre-training/domain-specific fine-tuning paradigm will be highly effective for a variety of data-scarce vision problems." ^[1] Nearly every CNN-based vision system released between 2014 and 2020 followed the same recipe.

Stage 2: Domain-specific fine-tuning

The pre-trained CNN is then fine-tuned on the target detection dataset (e.g., PASCAL VOC trainval) ^[1]. The 1,000-way ImageNet classifier head is replaced with a randomly initialized (N+1)-way classifier (N object classes plus background; N = 20 for PASCAL VOC, so the new head is 21-way) ^[1]. Fine-tuning treats Selective Search proposals as training images. A proposal with intersection-over-union (IoU) of at least 0.5 with any ground-truth box of class C is labeled as a positive for class C; proposals with IoU less than 0.5 with all ground-truth boxes are labeled as background ^[1]. Mini-batches of size 128 are built from 32 positive windows (across all classes) and 96 background windows ^[1]. Stochastic gradient descent runs at a learning rate of 0.001, one tenth of the initial pre-training rate, to avoid overwriting the pre-trained weights ^[1].

Stage 3: Per-class linear SVMs

After fine-tuning, the (N+1)-way classifier is discarded and the 4,096-d fc7 features become the SVM inputs ^[1]. SVM training uses a different labeling rule than fine-tuning, which the paper acknowledges as a quirk: positives are only the ground-truth boxes themselves; negatives are proposals with IoU less than 0.3 to any ground-truth box; proposals with IoU between 0.3 and 0.5 are ignored ^[1]. The 0.3 threshold was chosen by grid search; setting it to 0.5 reduced mAP by 5 points and setting it to 0 reduced mAP by 4 points ^[1]. To handle the very large number of negatives, the paper uses hard negative mining: a few rounds of training and re-evaluation, each round adding the most-confidently-misclassified negatives to the training set ^[1][5]. Hard negative mining converges quickly; the paper notes that mAP stops increasing after a single pass over all images ^[1].

Stage 4: Per-class bounding-box regressors

A per-class linear regressor is trained on pool5 features to predict the four bounding-box offsets ^[1]. The regressor uses ridge regression with a regularization strength of 1,000 and only sees proposals with IoU of at least 0.6 with a ground-truth box ^[1].

Storage requirements

A practical curiosity of R-CNN training is the disk-space cost. Because the CNN is treated as a fixed feature extractor for SVM and regressor training, the 4,096-d features for every proposal in the training set must be stored ^[1]. For PASCAL VOC trainval the cache occupies tens of gigabytes; for the full ILSVRC 2013 detection set, it can reach hundreds of gigabytes ^[1]. This was one of the explicit motivations for the end-to-end design of Fast R-CNN, which never materializes per-region features outside the GPU ^[8].

How accurate is R-CNN?

The R-CNN paper reported state-of-the-art numbers on three benchmarks: PASCAL VOC 2010, PASCAL VOC 2012, and ILSVRC 2013 detection ^[1]. Exact numbers depend on the CNN backbone (AlexNet vs. VGG-16) and on whether bounding-box regression is used.

PASCAL VOC

Scroll sideways for more →

The headline 53.3% mAP on PASCAL VOC 2012 was a more than 30% relative improvement over the published state of the art ^[1]. With the deeper VGG-16 backbone, the same pipeline reached 66.0% on VOC 2007, a number that would have seemed implausible only a year earlier ^[1][6].

ILSVRC 2013 detection

R-CNN won the 2013 ImageNet 200-class detection challenge by a large margin ^[1].

Scroll sideways for more →

The 7.1-point gap to OverFeat was the largest in the challenge ^[1].

Inference time

R-CNN's accuracy gains came at a steep computational cost. Test-time numbers per image (including Selective Search and CNN feature extraction) ^[1]:

Scroll sideways for more →

The per-class SVM and regressor steps are negligible compared with the CNN forward pass. The cost is dominated by running the CNN once per proposal, roughly 2,000 times per image, with no shared computation ^[1][8]. Every successor in the R-CNN family attacked this bottleneck.

Why is R-CNN slow, and what are its limitations?

R-CNN was a research breakthrough but a pragmatic mess. The paper itself is candid about the limitations ^[1].

1. Slow inference. Running the CNN independently on roughly 2,000 proposals per image is the dominant cost. With VGG-16 it takes around 47 seconds per image on a K40 GPU, which rules out real-time use ^[1].
2. Slow, multi-stage training. Pretrain, fine-tune, train SVMs, then train regressors. Each stage requires the previous stage's output, so they cannot be parallelized; full training takes days ^[1][8].
3. Disk-heavy training. Cached fc7 features for every Selective Search proposal in the training set occupy hundreds of gigabytes for ILSVRC 2013 detection ^[1].
4. Selective Search is fixed. It runs on the CPU and is not learned, so the system is bottlenecked at the proposal stage and cannot adapt the proposals to the downstream task ^[3][9].
5. Inconsistent labeling rules. Fine-tuning treats proposals with IoU greater than 0.5 as positives, while SVM training uses only ground-truth boxes as positives and IoU less than 0.3 as negatives ^[1].
6. Warped inputs lose information. Anisotropic warping of arbitrary-aspect-ratio crops to 227 by 227 throws away geometric information that a more careful pooling strategy could preserve ^[1][7].

Every one of these limitations was addressed by a subsequent paper in the R-CNN family ^{[7][8][9][10]}.

What is the R-CNN family of models?

R-CNN started a research lineage in which each follow-up removed a bottleneck while preserving the two-stage structure of "propose, then classify and regress."

Scroll sideways for more →

How do Fast R-CNN and Faster R-CNN improve on it?

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun's SPPnet at ECCV 2014 asked the obvious question: why run the CNN 2,000 times when you could run it once? Their answer was a spatial pyramid pooling layer that pooled features from arbitrary rectangular regions of a single shared feature map at multiple scales, producing a fixed-length descriptor per proposal ^[7]. SPPnet was 24 to 102 times faster than R-CNN at test time with similar or better accuracy on PASCAL VOC 2007 ^[7].

Girshick's Fast R-CNN at ICCV 2015 generalized SPPnet and made the whole detector end-to-end trainable ^[8]. RoI Pooling divided each proposal's feature region into a fixed grid (typically 7 by 7) and max-pooled within each cell ^[8]. The pooled feature was fed through two fully connected layers, then split into a softmax classifier (K+1 classes) and a per-class bounding-box regressor, all trained with a single multi-task loss ^[8]. Fast R-CNN was 9x faster to train and 213x faster at test time than R-CNN with the same VGG-16 backbone ^[8]. The remaining bottleneck was Selective Search, which still ran on the CPU.

Faster R-CNN by Ren, He, Girshick, and Sun (NeurIPS 2015) removed that last external step with a Region Proposal Network, a small fully convolutional sub-network that ran on the shared feature map and predicted candidate boxes directly using anchors of multiple scales and aspect ratios ^[9]. Because the RPN ran on the GPU and shared most of its computation with the detector, proposals became almost free: about 10 ms per image instead of 2 seconds ^[9]. The full system reached 5 fps with VGG-16 on a K40 GPU and 73.2% mAP on PASCAL VOC 2007 ^[9].

What are Mask R-CNN, Cascade R-CNN, and beyond?

Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick extended Faster R-CNN in 2017 with a third output branch that produced a binary segmentation mask for each region of interest ^[10]. Mask R-CNN also replaced RoI Pooling with RoIAlign, a bilinear-interpolation crop that avoided the integer rounding of RoI Pool and substantially improved mask quality ^[10]. It won the ICCV 2017 Best Paper Award ^[10]. R-FCN (Dai et al., NeurIPS 2016) used position-sensitive score maps to remove the per-RoI fully connected layers. Cascade R-CNN (Cai and Vasconcelos, CVPR 2018) chained multiple heads trained at progressively higher IoU thresholds, lifting COCO AP by about 4 points ^[11]. Sparse R-CNN, Dynamic Head, HTC, and Light-Head R-CNN continued to refine the architecture into the early 2020s.

How does R-CNN compare with single-stage detectors?

From around 2016 onward a parallel line of one-stage detectors emerged, trading a few mAP points for higher frame rates by skipping the explicit proposal step.

Scroll sideways for more →

Two-stage detectors of the R-CNN family generally win on accuracy in cluttered or small-object regimes; one-stage detectors win on speed and simplicity ^[13][14][15]. From 2020 the transformer-based DETR family began to displace both for cutting-edge research, although Faster R-CNN and Mask R-CNN remain widely used as transfer-learning starting points ^[16].

How can I use R-CNN today?

The original R-CNN code was released by Ross Girshick on GitHub in May 2014 under the Simplified BSD License at github.com/rbgirshick/rcnn ^[19]. It is implemented in MATLAB on top of an early version of Caffe, the deep learning framework developed at UC Berkeley by Yangqing Jia and others ^[19]. The repository requires Caffe v0.999 and is no longer actively maintained, but it is preserved as the historical artifact for the CVPR 2014 paper and the extended TPAMI 2016 journal version ^[2][19].

Scroll sideways for more →

For practitioners who want R-CNN-style object detection today, Faster R-CNN with a ResNet-50-FPN backbone in Detectron2 or torchvision is the closest spiritual descendant of the 2014 system, and it runs hundreds of times faster while reaching about twice the COCO accuracy ^[9][10].

Why was R-CNN so influential?

R-CNN had an immediate and lasting effect on computer vision research. It was the first paper to convincingly demonstrate that the CNN revolution from ImageNet 2012 would carry over to object detection, and it set the template, backbone CNN + region proposals + per-class classifier head, that defined two-stage detection for the rest of the decade ^[1]. The paper has over 30,000 Google Scholar citations ^[20], won the 2024 Longuet-Higgins Prize from the IEEE Computer Society ^[21], and inspired at least five major direct-descendant papers (SPPnet, Fast R-CNN, Faster R-CNN, Mask R-CNN, Cascade R-CNN) ^{[7][8][9][10][11]}. It also established the "pretrain on ImageNet, fine-tune on the target task" transfer-learning recipe as a default for vision tasks for nearly a decade ^[1].

By 2017 R-CNN itself had largely been displaced in production by Faster R-CNN and Mask R-CNN, which were faster and more accurate ^[9][10]. From around 2020 the transformer-based DETR family began to take over the cutting-edge benchmarks ^[16]. Even so, R-CNN remains the historical reference point for deep-learning object detection. Surveys routinely begin with R-CNN as the dividing line between the hand-crafted-feature era (HOG, DPM, Selective Search + Bag of Words) and the modern CNN-based era. The two key insights of the paper, that bottom-up region proposals plus high-capacity CNN features could outperform sliding-window methods, and that ImageNet pre-training plus task-specific fine-tuning could close the data gap for detection, are still widely used over a decade later ^[1].

See also

• Object detection
• Convolutional neural network
• AlexNet
• VGGNet
• Faster R-CNN
• Mask R-CNN
• YOLO (object detection)
• SSD (Single Shot MultiBox Detector)
• PASCAL VOC
• Instance segmentation

References

1. Girshick, R., Donahue, J., Darrell, T., Malik, J. (2014). "Rich feature hierarchies for accurate object detection and semantic segmentation". *CVPR 2014*: 580-587. arXiv:1311.2524 (first posted 11 November 2013, v5 posted 22 October 2014). https://arxiv.org/abs/1311.2524. ↩
2. Girshick, R., Donahue, J., Darrell, T., Malik, J. (2016). "Region-based Convolutional Networks for Accurate Object Detection and Segmentation". *IEEE TPAMI* 38(1): 142-158. The extended journal version of the 2014 CVPR paper. ↩
3. Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., Smeulders, A. W. M. (2013). "Selective Search for Object Recognition". *IJCV* 104(2): 154-171. ↩
4. Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012). "ImageNet Classification with Deep Convolutional Neural Networks" (AlexNet). *NeurIPS* 25: 1097-1105. ↩
5. Felzenszwalb, P. F., Girshick, R. B., McAllester, D., Ramanan, D. (2010). "Object Detection with Discriminatively Trained Part Based Models". *IEEE TPAMI* 32(9): 1627-1645. ↩
6. Simonyan, K., Zisserman, A. (2014). "Very Deep Convolutional Networks for Large-Scale Image Recognition" (VGGNet). arXiv:1409.1556. ↩
7. He, K., Zhang, X., Ren, S., Sun, J. (2014). "Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition" (SPPnet). *ECCV 2014*: 346-361. arXiv:1406.4729. ↩
8. Girshick, R. (2015). "Fast R-CNN". *ICCV 2015*: 1440-1448. arXiv:1504.08083. ↩
9. Ren, S., He, K., Girshick, R., Sun, J. (2015). "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks". *NeurIPS* 28: 91-99. arXiv:1506.01497. ↩
10. He, K., Gkioxari, G., Dollar, P., Girshick, R. (2017). "Mask R-CNN". *ICCV 2017* (Best Paper Award). arXiv:1703.06870. ↩
11. Cai, Z., Vasconcelos, N. (2018). "Cascade R-CNN: Delving into High Quality Object Detection". *CVPR 2018*. arXiv:1712.00726. ↩
12. Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S. (2017). "Feature Pyramid Networks for Object Detection". *CVPR 2017*. arXiv:1612.03144.
13. Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollar, P. (2017). "Focal Loss for Dense Object Detection" (RetinaNet). *ICCV 2017*. arXiv:1708.02002. ↩
14. Redmon, J., Divvala, S., Girshick, R., Farhadi, A. (2016). "You Only Look Once: Unified, Real-Time Object Detection" (YOLO). *CVPR 2016*. arXiv:1506.02640. ↩
15. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A. C. (2016). "SSD: Single Shot MultiBox Detector". *ECCV 2016*. arXiv:1512.02325. ↩
16. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S. (2020). "End-to-End Object Detection with Transformers" (DETR). *ECCV 2020*. arXiv:2005.12872. ↩
17. Lin, T.-Y., et al. (2014). "Microsoft COCO: Common Objects in Context". *ECCV 2014*. arXiv:1405.0312.
18. Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., Zisserman, A. (2010). "The PASCAL Visual Object Classes (VOC) Challenge". *IJCV* 88(2): 303-338.
19. Girshick, R. rcnn (original implementation). https://github.com/rbgirshick/rcnn. ↩
20. Region Based Convolutional Neural Networks. Wikipedia. https://en.wikipedia.org/wiki/Region_Based_Convolutional_Neural_Networks. ↩
21. Girshick, R. "Personal page, publication list and awards". https://www.rossgirshick.info/. ↩