Machine learning terms/Computer Vision

See also: Machine learning terms

The key machine learning terms for computer vision describe how neural networks turn pixels into predictions: a convolution slides a small learned filter (kernel) across an image to detect patterns, a convolutional neural network (CNN) stacks many such filters with pooling to build hierarchical features, and downstream tasks include image recognition (labeling a whole image), object detection (drawing bounding boxes around objects), and segmentation (labeling every pixel). Detector quality is scored with intersection over union (IoU), the area of overlap divided by the area of union between a predicted box and the ground truth.^[21][24] Training is made robust with data augmentation, and CNNs gain efficiency from translational invariance, convolutional filters shared across the image, and depthwise separable convolutions.^[22][23]

Computer vision (CV) is the subfield of artificial intelligence and machine learning that builds systems capable of extracting information from digital images, video, and other visual inputs. The discipline aims to give machines the ability to identify objects, recognize people, understand scenes, infer geometry, and reason about events. Modern computer vision has converged with deep learning, and most state of the art systems are built on convolutional neural networks, vision transformers, and multimodal foundation models trained on hundreds of millions of image text pairs.

The field traces its origins to early experiments at MIT in the 1960s, including the 1966 Summer Vision Project led by Seymour Papert, which famously underestimated the difficulty of human level visual perception. Six decades of research have produced techniques ranging from hand crafted detectors such as SIFT and HOG to billion parameter foundation models such as CLIP, SAM, and DINOv2. Computer vision now powers face unlock, autonomous driving, medical imaging diagnostics, satellite analysis, content moderation, augmented reality, and generative tools that turn text prompts into photorealistic images.

What are the key machine learning computer vision terms?

This section defines the core vocabulary of computer vision as it appears in a machine learning glossary, with each term cross linked to its dedicated article. The definitions follow primary sources including Google's Machine Learning glossary and the textbook Deep Learning by Goodfellow, Bengio, and Courville (MIT Press, 2016).^[21][22] Each concept also appears in the more detailed sections later in this page.

Scroll sideways for more →

What is a convolution?

A convolution is the core mathematical operation of a CNN. Google's Machine Learning glossary defines it as a process that "mixes the convolutional filter and the input matrix in order to train weights."^[21] In practice the operation has two steps: an element wise multiplication of the convolutional filter and a slice of the input matrix, followed by a summation of every value in the resulting product matrix.^[21] Repeating this across the whole image produces a feature map. A convolutional filter, also called a kernel, is "a matrix having the same rank as the input matrix, but a smaller shape" (commonly 3x3 or 5x5), and the same filter is reused at every position, a property known as weight sharing.^[21] The stride sets how many pixels the filter moves between applications, which controls how much the output is downsampled.

What is a convolutional neural network?

A convolutional neural network (CNN) is, in Google's words, "a neural network in which at least one layer is a convolutional layer," typically combining convolutional layers, pooling layers, and dense (fully connected) layers.^[21] CNNs became the dominant approach for vision in 2012, when AlexNet (Krizhevsky, Sutskever, and Hinton) cut the ImageNet top 5 error rate from the prior best of 26.2 percent to 15.3 percent, a 10.8 point jump that ended the era of hand crafted features.^[1] Goodfellow, Bengio, and Courville explain that convolution improves a machine learning system through three ideas: "sparse interactions, parameter sharing and equivariant representations."^[22] Sparse interactions come from small kernels touching only a local region, parameter sharing reuses one kernel everywhere, and equivariance to translation means a shifted input produces a correspondingly shifted output.^[22] Stacking small filters deeply, as in VGG-16, lets the network learn edges, then textures, then object parts, then whole objects.

What is pooling and translational invariance?

Pooling is a reduction operation that summarizes a small spatial region of a feature map into a single value, most commonly with max pooling or average pooling. Beyond shrinking resolution, pooling builds robustness to position: Goodfellow, Bengio, and Courville note that "pooling helps to make the representation approximately invariant to small translations of the input."^[22] This is the basis of translational invariance, the ability to recognize an object no matter where it sits in the frame. Related properties include rotational invariance (robustness to in plane rotation) and size invariance (robustness to apparent object size), which are usually encouraged through architecture choices and data augmentation rather than guaranteed by pooling alone.

What is image recognition and object detection?

Image recognition, also called image classification, assigns one or more labels to a whole image. The canonical benchmark is ImageNet ILSVRC, which contains 1,000 categories and 1,281,167 training images (about 1.28 million), with top 1 accuracy rising from roughly 56 percent for AlexNet in 2012 to over 91 percent for recent models.^[16] Object detection goes further by localizing and classifying multiple objects at once, returning a bounding box and a class label with a confidence score for each. Google's glossary defines a bounding box simply as "the (x, y) coordinates of a rectangle around an area of interest."^[21] The standard detection benchmark is MS COCO (2014), with 80 object categories and over 200,000 labeled images.^[15] Detection quality is reported as mean average precision (mAP), computed across a range of IoU thresholds.

What is intersection over union?

Intersection over union (IoU), sometimes called the Jaccard index, is the standard geometric metric for object detection and segmentation. It is defined as the area of overlap between a predicted box and the ground truth box divided by the area of their union: IoU = area of overlap / area of union.^[24] The value ranges from 0 (no overlap) to 1 (a perfect match), and a detection is usually counted as correct when its IoU exceeds a threshold such as 0.50.^[24] The COCO primary metric averages mAP over IoU thresholds from 0.50 to 0.95 in steps of 0.05, which is much stricter than the Pascal VOC convention of a single 0.50 threshold.^[15] IoU also drives non maximum suppression, which removes duplicate boxes that overlap too heavily.

What is data augmentation?

Data augmentation is, per Google's glossary, "artificially boosting the range and number of training examples by transforming existing examples to create additional examples."^[21] Typical transforms for images include horizontal flips, random crops, rotations, color jitter, and mixing strategies such as Mixup and CutMix. Augmentation enlarges the effective training set without new labels, reduces overfitting, and teaches the model the invariances (position, scale, lighting) that a classifier should ignore. It is standard practice in nearly every modern vision training pipeline and is especially valuable when labeled data is scarce.

What is a depthwise separable convolution?

A depthwise separable convolution (sepCNN) factorizes a standard convolution into two cheaper steps: a depthwise convolution that filters each input channel independently, followed by a pointwise (1x1) convolution that combines channels. Introduced for mobile vision in MobileNets (Howard et al., 2017), this factorization costs roughly 8 to 9 times less computation than a standard 3x3 convolution at the price of a small accuracy reduction.^[23] The technique underpins efficient architectures such as MobileNet and Xception and is one reason CNNs remain competitive on phones and edge accelerators where compute and memory are tightly constrained.

history

Computer vision history divides into several broad eras, each defined by the dominant image representation and algorithms.

Scroll sideways for more →

The transition to deep learning dates to the 2012 ImageNet Challenge, where AlexNet (Krizhevsky, Sutskever, Hinton) cut top 5 error from 26.2 to 15.3 percent, ending the era of hand crafted features almost overnight.^[1]

core tasks

Computer vision is organized around a small number of canonical tasks. Real applications often combine several of them, but treating each as a benchmark has driven steady progress.

image classification

Image classification assigns one or more labels to a whole image. The classic benchmark is ImageNet with 1,000 categories and 1.28 million training images.^[16] Top 1 accuracy rose from about 56 percent for AlexNet in 2012 to over 91 percent for modern models such as CoCa and EVA-02 by 2024.

object detection

Object detection localizes and classifies multiple objects, producing bounding boxes with class labels and confidence scores. It is evaluated using intersection over union thresholds and mean average precision (mAP). The standard benchmark is MS COCO (2014), with 80 categories and over 200,000 labeled images.^[15]

image segmentation

Segmentation assigns a label to every pixel. Three main variants exist:

Scroll sideways for more →

The distinction between things (countable objects) and stuff (amorphous regions like sky) was formalized by Kirillov and colleagues in 2019.

keypoint and pose estimation

Keypoint detection identifies landmarks such as eye corners or body joints. Keypoints and landmarks are used in face alignment, hand tracking, animal pose, and motion capture. Notable systems include OpenPose (2017) and MediaPipe (2019).

depth estimation

Depth estimation predicts pixel wise distance from the camera. It can use stereo pairs, structured light, time of flight sensors, or a single monocular image. Modern monocular models such as MiDaS (2020) and Depth Anything (2024) generalize across domains.

optical flow

Optical flow estimates per pixel motion between video frames. Horn and Schunck (1981) and Lucas and Kanade (1981) defined the classical formulation. Deep approaches like FlowNet (2015), PWC-Net (2018), and RAFT (2020) lead modern benchmarks (Sintel, KITTI).

other tasks

Additional tasks include image generation, image to image translation, super resolution, denoising, image captioning, visual question answering, action recognition, 3D reconstruction, and visual SLAM.

classical computer vision

Before deep learning, computer vision relied on hand designed detectors and descriptors. Many remain useful today for limited data or strict latency budgets, and they are essential to robotics and structure from motion pipelines.

Scroll sideways for more →

Other classical operations include Sobel and Prewitt gradients, Laplacian of Gaussian blob detection, RANSAC for robust fitting, and the Hough transform for parametric shape detection.

the convolutional neural network era

The modern era began with convolutional neural networks (CNNs). CNNs stack convolutional layers, ReLU nonlinearities, pooling, and fully connected layers to learn hierarchical features from pixels. Key ideas include weight sharing, translational invariance, and stacking small convolutional filters deeply.

Scroll sideways for more →

Convolutions remain dominant on mobile and edge devices due to hardware efficiency. Pooling and stride parameters control how spatial information is downsampled.

object detection architectures

Object detection has been one of the most active subfields since 2014. Detectors are typically classified as two stage, single stage, or transformer based.

Scroll sideways for more →

RetinaNet introduced focal loss to address foreground/background imbalance, hitting 39.1 mAP on COCO in 2017. Modern transformer detectors such as Co-DETR with Swin-Large push above 66 mAP.^[20] The YOLO family, originated by Joseph Redmon in 2016, has been continued by Ultralytics, Meituan (YOLOv6), and Tsinghua researchers (YOLOv8+), and is widely deployed in real time applications.

segmentation architectures

Scroll sideways for more →

U-Net remains the baseline for medical image segmentation and is the backbone of the denoiser in latent diffusion models such as Stable Diffusion.^[5][10]

vision transformers

The Vision Transformer paper by Dosovitskiy et al. at Google Brain (2020) showed that the transformer architecture, designed for language, could match or beat CNNs on classification given enough pretraining data.^[7] ViT splits the image into patches (typically 16x16), embeds each linearly, adds positional embeddings, and processes the sequence with a standard transformer encoder.^[7]

Scroll sideways for more →

Vision transformers benefit greatly from pretraining on large datasets such as JFT-300M (Google, 300M images) or LAION-2B (Stable Diffusion, 2B image text pairs).

multimodal vision language models

Multimodal models aligning images and text in a shared embedding space have transformed the field since 2021. They enable zero shot classification, open vocabulary detection, image text retrieval, and serve as the visual front end of multimodal LLMs such as GPT-4o, Gemini, and Claude.^[9]

Scroll sideways for more →

Text to image generation typically uses classifier free guidance, DDPM, and increasingly transformer based diffusion (DiT). Video extensions like Sora (OpenAI, 2024) and Veo (Google DeepMind, 2024) generate multi second clips using spacetime patches as tokens.

3d vision

3D computer vision recovers geometric structure from images. Classical pipelines combine multi view stereo, structure from motion (SfM), and bundle adjustment. Modern systems learn 3D representations end to end.

Scroll sideways for more →

Depth estimation, photometric stereo, structured light, and time of flight sensors complement learned methods in phones, vehicles, and AR headsets like the Apple Vision Pro and Meta Quest 3.

visual foundation models

Foundation models are large pretrained models that adapt to many downstream tasks. The vision community now has several visual and vision language foundation models, often with permissive or open licenses.

Scroll sideways for more →

These models are used as drop in feature extractors. DINOv2 ViT-L/14 features exceed 86 percent top 1 ImageNet accuracy with a linear probe, and SAM is widely used in data augmentation and labeling pipelines.^[14][13]

datasets

Progress is closely tied to large labeled datasets. The most influential are listed below.

Scroll sideways for more →

Dataset bias and licensing are persistent concerns. ImageNet's person categories were partly retired in 2019, and LAION-5B was briefly withdrawn in 2023 over problematic content that has since been removed.

hardware

Model scale is bounded by available training and inference hardware.

Scroll sideways for more →

Quantization, pruning, and knowledge distillation fit large models onto edge accelerators. Frameworks like Core ML, TensorRT, OpenVINO, and ONNX Runtime translate models into device specific instructions.

benchmarks and metrics

Scroll sideways for more →

Intersection over union is the standard geometric metric. The COCO mAP averaged over IoU 0.50 to 0.95 in steps of 0.05 is much stricter than Pascal VOC at IoU 0.50.

key concepts

Scroll sideways for more →

applications

Computer vision is embedded in many consumer and industrial products.

Scroll sideways for more →

challenges and open problems

Scroll sideways for more →

related concepts in this wiki

The following terms have their own articles, preserved from the original index.

• augmented reality
• bounding box
• convolution
• convolutional filter
• convolutional layer
• convolutional neural network
• convolutional operation
• data augmentation
• depthwise separable convolutional neural network (sepCNN)
• downsampling
• image recognition
• intersection over union (IoU)
• keypoints
• landmarks
• MNIST
• pooling
• rotational invariance
• size invariance
• spatial pooling
• stride
• subsampling
• translational invariance

see also

• Machine learning
• Deep learning
• Convolutional neural network
• Vision transformer
• Generative adversarial network
• Diffusion model
• Multimodal learning

references

1. Krizhevsky, Sutskever, Hinton (2012). *ImageNet Classification with Deep CNNs*. NeurIPS. ↩
2. He, Zhang, Ren, Sun (2016). *Deep Residual Learning for Image Recognition*. CVPR.
3. Ren, He, Girshick, Sun (2015). *Faster R-CNN*. NeurIPS.
4. Long, Shelhamer, Darrell (2015). *Fully Convolutional Networks for Semantic Segmentation*. CVPR.
5. Ronneberger, Fischer, Brox (2015). *U-Net*. MICCAI. ↩
6. He, Gkioxari, Dollar, Girshick (2017). *Mask R-CNN*. ICCV.
7. Dosovitskiy et al. (2021). *An Image is Worth 16x16 Words*. ICLR. ↩
8. Liu et al. (2021). *Swin Transformer*. ICCV.
9. Radford et al. (2021). *Learning Transferable Visual Models From Natural Language Supervision*. ICML. ↩
10. Rombach et al. (2022). *High-Resolution Image Synthesis with Latent Diffusion Models*. CVPR. ↩
11. Mildenhall et al. (2020). *NeRF*. ECCV.
12. Kerbl et al. (2023). *3D Gaussian Splatting*. ACM TOG.
13. Kirillov et al. (2023). *Segment Anything*. ICCV. ↩
14. Oquab et al. (2024). *DINOv2*. TMLR. ↩
15. Lin et al. (2014). *Microsoft COCO*. ECCV. ↩
16. Russakovsky et al. (2015). *ImageNet Large Scale Visual Recognition Challenge*. IJCV. ↩
17. Lowe (2004). *Distinctive Image Features from Scale-Invariant Keypoints*. IJCV.
18. Dalal and Triggs (2005). *Histograms of Oriented Gradients for Human Detection*. CVPR.
19. LeCun, Bottou, Bengio, Haffner (1998). *Gradient-Based Learning Applied to Document Recognition*. Proc. IEEE.
20. Carion et al. (2020). *End-to-End Object Detection with Transformers*. ECCV. ↩
21. Google. *Machine Learning Glossary: Image* (convolution, convolutional filter, convolutional layer, convolutional neural network, convolutional operation, bounding box, data augmentation). developers.google.com/machine-learning/glossary/image. ↩
22. Goodfellow, Bengio, Courville (2016). *Deep Learning*, Chapter 9 (Convolutional Networks). MIT Press. ↩
23. Howard et al. (2017). *MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications*. arXiv:1704.04861. ↩
24. Rezatofighi et al. (2019). *Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression*. CVPR. ↩