Stride
Last reviewed
Sources
11 citations
Review status
Source-backed
Revision
v6 ยท 2,954 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
11 citations
Review status
Source-backed
Revision
v6 ยท 2,954 words
Add missing citations, update stale details, or suggest a clearer explanation.
Stride is the step size by which a filter (or pooling window) moves across the input in a convolutional neural network (CNN): a stride of 1 shifts the filter one position at a time and visits every location, while a stride greater than 1 skips positions and downsamples the output. Stride is the hyperparameter that, together with kernel size and padding, determines the spatial dimensions of the output feature map, and it directly trades spatial resolution against computational cost. The textbook "Dive into Deep Learning" defines it concisely as "the number of rows and columns traversed per slide." [7]
Stride applies to both the convolution operation and to pooling. It plays a central role in balancing computational efficiency, spatial resolution, and the network's ability to capture patterns at different scales. [9]
In the context of a convolutional layer, a filter (also called a kernel) slides over the input data to compute dot products at each position. The stride specifies the number of positions the filter shifts between successive applications. A stride of 1 means the filter moves one pixel at a time, covering every possible position. A stride of 2 means the filter jumps two pixels at each step, skipping every other position. [7]
Stride can be specified as a single integer (applied equally in both the horizontal and vertical directions) or as a tuple such as (2, 2) or (1, 2) to allow different stride values along each spatial dimension. Most frameworks default to a stride of 1 when no value is explicitly provided. [10]
Consider a 7x7 input with a 3x3 filter. With stride 1, the filter can be placed at 5 positions along each axis, producing a 5x5 output. With stride 2, the filter lands at positions 0, 2, and 4 along each axis, producing a 3x3 output. The larger stride reduces the output size because the filter visits fewer positions.
The relationship between stride and output dimensions is governed by a well-known formula. For a single spatial dimension, the output size is:
O = floor((W - K + 2P) / S) + 1
Where:
For two-dimensional inputs, the formula is applied independently to each spatial dimension: [7]
When dilation is used, the formula generalizes to: [3]
O = floor((W + 2P - D * (K - 1) - 1) / S) + 1
where D is the dilation rate.
| Input Size | Kernel Size | Padding | Stride | Output Size |
|---|---|---|---|---|
| 32x32 | 3x3 | 0 | 1 | 30x30 |
| 32x32 | 3x3 | 1 | 1 | 32x32 |
| 32x32 | 3x3 | 1 | 2 | 16x16 |
| 224x224 | 7x7 | 3 | 2 | 112x112 |
| 8x8 | 3x3 | 0 | 3 | 2x2 |
The table illustrates that stride 1 with appropriate padding preserves the input dimensions, while stride 2 halves them. This is the most common configuration in modern CNN architectures. The 224x224 to 112x112 row is exactly the input stem used by ResNet, whose first layer is a 7x7 convolution with stride 2. [4]
The two most frequently used stride values are 1 and 2. They serve fundamentally different purposes in a network's design.
With stride 1 and "same" padding (P = (K - 1) / 2 for odd kernel sizes), the output feature map retains the same spatial dimensions as the input. This is the default setting in most convolutional layers and is preferred when fine-grained spatial detail must be preserved. Early layers in a network and layers within residual blocks commonly use stride 1. [10]
Stride 2 roughly halves the spatial dimensions of the input, serving as a form of downsampling. Because the output has roughly half the width and half the height, it contains about one quarter as many elements, reducing the number of activations in the feature map by approximately 75% and significantly lowering the computational cost and memory usage of subsequent layers. Stride 2 is commonly used at transition points in a network where spatial resolution is intentionally reduced.
Networks such as VGG used max pooling with stride 2 to downsample [6], while more recent architectures like ResNet and many others use strided convolutions for the same purpose. [4]
Stride is also a parameter in pooling layers, including max pooling and average pooling. In pooling, the stride controls how the pooling window moves across the feature map. The same output size formula applies. [7]
In many standard pooling configurations, the stride equals the pool size (for example, a 2x2 max pooling window with stride 2), which produces non-overlapping pooling regions and halves the spatial dimensions. When the stride is smaller than the pool size, overlapping pooling occurs. AlexNet famously used overlapping max pooling with a 3x3 window and stride 2. In the words of Krizhevsky, Sutskever, and Hinton, "This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2, z = 2," and they added that "models with overlapping pooling find it slightly more difficult to overfit." [5]
Traditionally, CNNs alternated convolutional layers with pooling layers to progressively reduce spatial dimensions. In 2014, Springenberg, Dosovitskiy, Brox, and Riedmiller published "Striving for Simplicity: The All Convolutional Net" (ICLR 2015 workshop), which demonstrated that max pooling layers could be entirely replaced by convolutional layers with stride 2. The paper's central claim is that "max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks," including CIFAR-10, CIFAR-100, and ImageNet. [1]
The key insight of this work is that pooling is a fixed, non-learnable operation, while a strided convolution is a learnable downsampling operation. By replacing pooling with strided convolution, the network gains additional trainable parameters that allow it to learn the optimal way to reduce spatial dimensions for the task at hand. The authors reported that "when pooling is replaced by an additional convolution layer with stride r=2 performance stabilizes and even improves on the base model." [1] Their all-convolutional network reached a test error of 9.08% on CIFAR-10 without data augmentation and 7.25% with augmentation, 33.71% on CIFAR-100, and a top-1 validation error of 41.2% on ImageNet. [1]
This approach has been widely adopted. Many modern architectures, including ResNet [4], DenseNet, and ConvNeXt, use strided convolutions for downsampling rather than pooling layers. ConvNeXt, for example, replaces the classic ResNet stem (a 7x7 convolution with stride 2 followed by max pooling, a 4x downsampling) with a single non-overlapping 4x4 convolution with stride 4, a "patchify" stem borrowed from Vision Transformers; this change nudged ImageNet top-1 accuracy from 79.4% to 79.5% in the authors' modernization study. [11]
| Approach | Type | Learnable | Parameters | Typical Use |
|---|---|---|---|---|
| Max pooling (stride 2) | Fixed operation | No | 0 | Classical architectures (VGG, AlexNet) |
| Average pooling (stride 2) | Fixed operation | No | 0 | Transition layers, global pooling |
| Strided convolution (stride 2) | Learned operation | Yes | K x K x C_in x C_out | Modern architectures (ResNet, ConvNeXt) |
Transposed convolutions (also called fractionally strided convolutions or sometimes incorrectly called deconvolutions) use stride in the opposite manner compared to standard convolutions. While stride greater than 1 in a standard convolution reduces spatial dimensions, stride greater than 1 in a transposed convolution increases spatial dimensions, performing upsampling. [3]
In a transposed convolution, the stride parameter controls how much spacing is inserted between input elements before the convolution is applied. A stride of 2 effectively doubles the spatial dimensions of the output. This is why the operation is also called "fractionally strided convolution": a stride of 2 over the output is equivalent to a stride of 1/2 over the input. [3]
Transposed convolutions with stride 2 are widely used in:
Transposed convolutions with certain stride and kernel size combinations can produce checkerboard artifacts in the output. Odena, Dumoulin, and Olah (2016) documented this issue in their influential Distill article "Deconvolution and Checkerboard Artifacts," explaining that "deconvolution has uneven overlap when the kernel size (the output window size) is not divisible by the stride." [2] As a remedy they proposed to "resize the image (using nearest-neighbor interpolation or bilinear interpolation) and then do a convolutional layer" (the "resize-convolution" approach). [2] Choosing kernel sizes that are divisible by the stride (for example, a 4x4 kernel with stride 2) also helps reduce these artifacts. [2]
The receptive field of a neuron in a CNN is the region of the input image that influences that neuron's activation. Stride has a significant effect on the receptive field of neurons in deeper layers.
When a layer uses stride greater than 1, neurons in subsequent layers effectively "see" a larger region of the original input. This is because each position in the downsampled feature map corresponds to a larger area of the input. Specifically, the receptive field size at layer l can be computed recursively:
r_l = r_(l-1) + (k_l - 1) * product of all strides in layers 1 to (l-1)
where r_l is the receptive field at layer l, k_l is the kernel size at layer l, and the product term accumulates all preceding stride values. A stride of 2 at any layer effectively doubles the contribution of all subsequent layers to the receptive field. [9]
This relationship means that using strided convolutions or strided pooling in early layers is an efficient way to rapidly increase the receptive field, allowing deeper neurons to capture large-scale patterns and global context in the input.
Different architectures use stride strategically at specific points in the network. The table below summarizes stride usage in well-known models.
| Architecture | Year | First Layer Stride | Downsampling Method | Notes |
|---|---|---|---|---|
| LeNet-5 | 1998 | 1 | Average pooling (stride 2) | Pioneer CNN for digit recognition |
| AlexNet | 2012 | 4 | Overlapping max pooling (stride 2) | Large initial stride to reduce the input; 60 million parameters [5] |
| VGG | 2014 | 1 | Max pooling (stride 2) | All conv layers use stride 1; pooling for downsampling [6] |
| GoogLeNet | 2014 | 2 | Mixed (pooling + strided conv) | Inception modules with stride 1 convolutions |
| ResNet | 2015 | 2 | Strided convolution (stride 2) | 7x7 conv with stride 2 at input; strided conv at transitions [4] |
| DenseNet | 2017 | 2 | Strided conv + pooling in transition layers | Dense blocks with stride 1 |
| EfficientNet | 2019 | 2 | Strided depthwise convolution | Compound scaling of depth, width, and resolution |
| ConvNeXt | 2022 | 4 | Strided convolution (stride 4 patchify stem) | Patchify stem inspired by Vision Transformers [11] |
AlexNet's first convolutional layer "filters the 224 x 224 x 3 input image with 96 kernels of size 11 x 11 x 3 with a stride of 4 pixels," the large stride being chosen to shrink the high-resolution input quickly. [5]
All major deep learning frameworks support stride as a parameter in convolution and pooling layers.
In PyTorch, the stride parameter is specified as an integer or tuple: [10]
import torch.nn as nn
# Stride 1 (default) - preserves spatial dimensions with padding
conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
# Stride 2 - halves spatial dimensions
conv2 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=2, padding=1)
# Asymmetric stride
conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 5), stride=(2, 1), padding=(1, 2))
# Transposed convolution with stride 2 for upsampling
deconv = nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=4, stride=2, padding=1)
In TensorFlow and Keras, the parameter is named strides (plural):
from tensorflow.keras.layers import Conv2D, Conv2DTranspose
# Stride 1 (default)
conv1 = Conv2D(filters=64, kernel_size=3, strides=1, padding='same')
# Stride 2 - halves spatial dimensions
conv2 = Conv2D(filters=128, kernel_size=3, strides=2, padding='same')
# Transposed convolution for upsampling
deconv = Conv2DTranspose(filters=64, kernel_size=4, strides=2, padding='same')
A key difference between the two frameworks is the default data format: PyTorch expects channel-first input (batch, channels, height, width), while TensorFlow/Keras defaults to channel-last input (batch, height, width, channels). The stride parameter behaves identically in both cases and is applied to the spatial dimensions.
Stride and dilation (also called atrous convolution) both affect how a convolution interacts with its input, but they serve different purposes. Stride controls how many positions the filter moves between applications, while dilation controls the spacing between filter elements. [3]
A dilated convolution with dilation rate d expands the receptive field without reducing the spatial resolution and without increasing the number of parameters. This is in contrast to stride, which increases the receptive field by reducing spatial resolution. Dilated convolutions are commonly used in models like DeepLab for semantic segmentation, where maintaining spatial resolution is important.
| Property | Stride > 1 | Dilation > 1 |
|---|---|---|
| Effect on output size | Reduces spatial dimensions | Preserves spatial dimensions |
| Effect on receptive field | Increases (indirectly) | Increases (directly) |
| Additional parameters | None (same kernel) | None (same kernel) |
| Primary use case | Downsampling, efficiency | Enlarging receptive field without downsampling |
When choosing a stride value for a particular layer, several factors should be considered:
Imagine you have a big picture made of tiny colored squares, like a mosaic. You want to look at the picture through a small magnifying glass that can only see a few squares at a time.
If the stride is 1, you move your magnifying glass one square at a time, looking at almost every spot on the picture. You get a very detailed idea of what the picture looks like, but it takes a long time.
If the stride is 2, you skip a square each time you move the magnifying glass. You finish looking at the picture much faster, but the picture you remember is smaller because you skipped some spots.
If the stride is 3, you skip even more squares. You finish very quickly, but you might miss some details.
So the stride is just how many squares you skip each time you move your magnifying glass. A small stride gives you more detail but takes more work. A big stride is faster but gives you a smaller, less detailed picture.