Depthwise separable convolutional neural network (sepCNN)

See also: Machine learning terms

Depthwise separable convolutional neural network (sepCNN)

A depthwise separable convolutional neural network (often abbreviated sepCNN) is a convolutional neural network that replaces standard convolution layers with depthwise separable convolutions. The depthwise separable convolution factors a single standard convolution into two smaller operations: a depthwise convolution that filters each input channel independently, followed by a pointwise convolution (a 1x1 convolution) that mixes the channels. This factorization keeps spatial filtering and channel mixing separate. It cuts both the parameter count and the number of multiply-add operations by roughly an order of magnitude for the kernel sizes used in typical computer vision networks, with only a small drop in accuracy on benchmarks such as ImageNet. The MobileNets paper reports that a network built from 3x3 depthwise separable convolutions "uses between 8 to 9 times less computation than standard convolutions at only a small reduction in accuracy" ^[3].

The most widely used sepCNN architectures are the MobileNet family, the Xception network, and the EfficientNet family. All three rely on depthwise separable convolutions as their main building block, and all three target settings where compute, memory, or energy is limited, such as edge AI and on-device inference. The original 1.0 MobileNet-224, for example, reaches 70.6% top-1 accuracy on ImageNet using just 569 million multiply-adds and 4.2 million parameters ^[3].

How does a depthwise separable convolution work?

A standard 2D convolution layer takes an input feature map of shape (D_F, D_F, M) and produces an output feature map of shape (D_F, D_F, N) using N filters of shape (D_K, D_K, M). Each output value is a sum across all M input channels and across the D_K by D_K spatial window. A depthwise separable convolution does the same job in two steps.

1. Depthwise convolution. Apply one D_K by D_K filter per input channel. There is no mixing across channels at this stage. The output has the same number of channels as the input.
2. Pointwise convolution. Apply N filters of shape 1 by 1 by M to the depthwise output. This step mixes the M channels into the N desired output channels but does no spatial filtering.

The two steps together do the same kind of work as a full convolution. They learn spatial patterns and they learn cross-channel combinations. They just learn them with different parameters instead of with one shared parameter tensor.

Cost comparison

The table below uses the notation from the MobileNets paper, where D_K is the kernel size, D_F is the feature map spatial size, M is the number of input channels, and N is the number of output channels.

Scroll sideways for more →

Dividing the depthwise separable cost by the standard cost gives a clean reduction ratio:

(D_K^2 * M * D_F^2 + M * N * D_F^2) / (D_K^2 * M * N * D_F^2) = 1/N + 1/D_K^2

For a typical layer with a 3x3 kernel and N = 64 output channels, this works out to about 1/64 + 1/9 ~= 0.127, so the depthwise separable version uses roughly 1/8 of the multiply-adds of a standard 3x3 convolution. For larger output widths the savings grow toward 1/9 (about 89% fewer operations) for 3x3 kernels, or 1/25 (about 96% fewer) for 5x5 kernels (Howard et al., 2017) ^[3].

Worked example

Consider a layer with 32 input channels, 64 output channels, a 3x3 kernel, and a 56x56 feature map.

Scroll sideways for more →

The parameter saving comes mostly from the fact that the depthwise step does not pay the M x N channel-mixing cost, and the pointwise step does not pay the D_K x D_K spatial cost.

Who invented depthwise separable convolutions?

The idea of separating spatial and channel mixing is older than its current name. Sifre and Mallat used separable convolutions in their 2013 work on rotation- and translation-invariant scattering for texture classification, which became the 2014 paper Rigid-Motion Scattering for Texture Classification (Sifre and Mallat, 2014) ^[1]. Sifre's PhD thesis at École Polytechnique generalized the construction further ^[2]. Around the same time, the original Inception V3 modules used factored 1xK and Kx1 convolutions, which are spatially separable in a different sense.

The deep learning community adopted the depthwise plus pointwise factorization in earnest with two papers in 2017. François Chollet's Xception paper (Chollet, 2017) framed the depthwise separable convolution as the limit of an Inception module with one tower per channel; in his words, "a depthwise separable convolution can be understood as an Inception module with a maximally large number of towers" ^[4]. Howard and colleagues at Google published MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications in the same year (Howard et al., 2017), which made the design popular for on-device computer vision and gave the abbreviation sepCNN much of its current weight ^[3].

Which architectures use depthwise separable convolutions?

Scroll sideways for more →

MobileNet v1 introduced two global hyperparameters that became standard practice: a width multiplier alpha that thins the channel count uniformly across all layers, and a resolution multiplier rho that shrinks the input image. Both let the same network template stretch from rich servers down to small embedded devices without redesign. The paper describes them as "two simple global hyper-parameters that efficiently trade off between latency and accuracy" ^[3].

MobileNet v2 wraps the depthwise separable block in an inverted residual with a linear bottleneck. The block first expands a thin tensor with a 1x1 convolution, runs a depthwise 3x3 in the expanded space, then projects back to a thin tensor with another 1x1 convolution. The final 1x1 has no ReLU on its output; Sandler et al. argue that "it is important to remove non-linearities in the narrow layers in order to maintain representational power" (Sandler et al., 2018) ^[5].

MobileNet v3 was discovered in part through hardware-aware neural architecture search (NAS), then refined by hand. It adds Squeeze-and-Excitation layers (Hu et al., 2018) ^[10] in some blocks and uses the hard-swish activation, a piecewise approximation of swish that is cheaper to evaluate on mobile CPUs (Howard et al., 2019) ^[6].

Xception keeps a more traditional shape with a stack of 36 depthwise separable convolutional layers wired together with residual connections. With essentially the same parameter budget as Inception V3 (about 22.9 million parameters), it slightly improves top-1 accuracy on ImageNet and shows a larger gain on an internal Google dataset of "350 million images and 17,000 classes" (Chollet, 2017) ^[4].

EfficientNet uses the MBConv block, which is a direct descendant of the MobileNet v2 inverted residual. The contribution of EfficientNet is compound scaling: instead of growing only depth, only width, or only input resolution, it scales all three together with a single coefficient. The EfficientNet-B0 baseline has about 5.3M parameters and 77.1% top-1 accuracy on ImageNet, while EfficientNet-B7 reaches 84.3%, a score it hits while being 8.4 times smaller and running 6.1 times faster at inference than the best previous ConvNet it was compared against (Tan and Le, 2019) ^[7].

How do you implement a depthwise separable convolution?

Most frameworks expose depthwise and pointwise convolutions as first-class operations.

Scroll sideways for more →

In PyTorch the depthwise step is implemented by the standard Conv2d layer with the groups argument set equal to the number of input channels. Each input channel then has its own filter, and the output channel count must be a multiple of groups.

What are depthwise separable convolutions used for?

Depthwise separable convolutions show up in almost every domain that wants high-quality vision under a tight compute or memory budget.

• On-device computer vision. MobileNet was designed for Android phones and powers many of the on-device vision features in Google services, including parts of Google Lens, smartphone cameras, and accessibility features that run without a network connection.
• Real-time object detection. Detection heads such as SSD-MobileNet and EfficientDet use sepCNN backbones to keep latency in the millisecond range on mobile chips and embedded boards.
• Real-time semantic segmentation. Mobile DeepLabv3, DeepLabv3+, and Lite R-ASPP all use depthwise separable convolutions in either the backbone or the segmentation head to stay within the budget of a phone or a microcontroller.
• Automotive and robotics perception. ADAS systems, drones, and warehouse robots commonly run sepCNN backbones because the inference cost can fit in a GPU-less system on chip without external accelerators.
• Backbone for downstream tasks. EfficientNet and MobileNet backbones are routinely fine-tuned for medical imaging, satellite imagery, and quality inspection in manufacturing, where the labeled datasets are small but inference must be cheap.

How do depthwise separable convolutions compare to other efficiency techniques?

Depthwise separable convolutions are one of several recurring tricks for shrinking CNNs. The table groups them by what they exploit.

Scroll sideways for more →

These methods compose well. EfficientNet, for example, layers depthwise separable MBConv blocks, Squeeze-and-Excitation, and compound scaling on top of one another. Production deployments often add quantization and sometimes pruning on top of an already efficient sepCNN backbone.

What are the limitations of depthwise separable convolutions?

The theoretical FLOP savings of depthwise separable convolutions do not always translate into proportional latency savings on real hardware.

• Lower arithmetic intensity. A depthwise convolution does very little arithmetic per byte of memory traffic compared to a dense convolution. On modern GPUs that are memory-bandwidth-limited for these layers, the practical speedup is often smaller than the FLOP ratio suggests.
• Lower expressive capacity per parameter. Splitting spatial and channel mixing constrains what each layer can learn. To reach the accuracy of a baseline ResNet, sepCNN designers sometimes have to compensate by making the network wider, deeper, or both, which eats into the savings.
• Sensitivity to normalization and activation. sepCNN blocks are usually paired with batch normalization and a nonlinearity such as ReLU, ReLU6, or hard-swish. Removing or changing these often hurts accuracy more than it does in dense CNNs.
• Optimizer sensitivity. Depthwise filters have very few parameters per channel, so initialization and learning-rate schedules that work for dense convolutions sometimes need adjustment.
• Hardware support gaps. Some older inference accelerators have well-tuned kernels for dense 3x3 convolutions but slow paths for depthwise convolutions, so a sepCNN backbone can run slower than a quantized dense baseline on those chips.

Are depthwise separable convolutions still used?

Depthwise separable convolutions remain the workhorse of efficient computer vision, but the broader landscape has shifted since 2020. Vision Transformers (ViT) (Dosovitskiy et al., 2021) split the image into patches and apply self-attention rather than convolution ^[13]. Pure ViTs scale impressively at the high end but are usually less efficient than sepCNN backbones in the small-model regime where MobileNet and EfficientNet live.

Hybrid models try to get both. MobileViT (Mehta and Rastegari, 2022) interleaves MobileNet-style depthwise separable blocks with small attention blocks ^[11]. EfficientFormer and Mobile-Former follow the same pattern. ConvNeXt (Liu et al., 2022) returns to a pure-convolution design but uses depthwise convolutions with large 7x7 kernels in a transformer-like macro layout, taking inspiration from the Swin Transformer ^[12]. EfficientNetV2 (Tan and Le, 2021) replaces the MBConv block with a Fused-MBConv in early stages, where the depthwise plus 1x1 expansion is collapsed back into a single 3x3 convolution, because on TPU and GPU the extra arithmetic intensity beats the FLOP savings ^[8].

In practice this means a vision model in 2025 is rarely only sepCNN, but it is also rarely no sepCNN. The factorization is a default tool, used where it pays off and dropped where it does not.

Explain like I'm 5 (ELI5)

Imagine you have a stack of colored transparency sheets, one red, one green, one blue, and you want to make a new picture out of them.

The normal way is to look at all three sheets at once, in every little square, and decide what the new color should be. That is a lot of looking.

The depthwise separable way is sneakier. First, look at each colored sheet on its own and decide what shape patterns are on it. Just shapes, no mixing. Then, in a second step, decide how to mix the red, green, and blue answers into the final color, but only for one tiny square at a time, with no shape work.

You get a similar picture, but you did far less staring. That is why phones use this trick to recognize cats, faces, and street signs without their batteries dying.

References

1. Sifre, L. and Mallat, S. (2014). *Rigid-Motion Scattering for Texture Classification*. arXiv:1403.1687. https://arxiv.org/abs/1403.1687 ↩
2. Sifre, L. (2014). *Rigid-Motion Scattering for Image Classification*. PhD Thesis, École Polytechnique. https://www.di.ens.fr/data/publications/papers/phd_sifre.pdf ↩
3. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). *MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications*. arXiv:1704.04861. https://arxiv.org/abs/1704.04861 ↩
4. Chollet, F. (2017). *Xception: Deep Learning with Depthwise Separable Convolutions*. CVPR 2017. https://arxiv.org/abs/1610.02357 ↩
5. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018). *MobileNetV2: Inverted Residuals and Linear Bottlenecks*. CVPR 2018. https://arxiv.org/abs/1801.04381 ↩
6. Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q. V., and Adam, H. (2019). *Searching for MobileNetV3*. ICCV 2019. https://arxiv.org/abs/1905.02244 ↩
7. Tan, M. and Le, Q. V. (2019). *EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks*. ICML 2019. https://arxiv.org/abs/1905.11946 ↩
8. Tan, M. and Le, Q. V. (2021). *EfficientNetV2: Smaller Models and Faster Training*. ICML 2021. https://arxiv.org/abs/2104.00298 ↩
9. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018). *ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices*. CVPR 2018. https://arxiv.org/abs/1707.01083 ↩
10. Hu, J., Shen, L., and Sun, G. (2018). *Squeeze-and-Excitation Networks*. CVPR 2018. https://arxiv.org/abs/1709.01507 ↩
11. Mehta, S. and Rastegari, M. (2022). *MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer*. ICLR 2022. https://arxiv.org/abs/2110.02178 ↩
12. Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). *A ConvNet for the 2020s*. CVPR 2022. https://arxiv.org/abs/2201.03545 ↩
13. Dosovitskiy, A., et al. (2021). *An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale*. ICLR 2021. https://arxiv.org/abs/2010.11929 ↩