# Depth

> Source: https://aiwiki.ai/wiki/depth
> Updated: 2026-05-11
> Categories: Machine Learning
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

*See also: [Machine learning terms](/wiki/machine_learning_terms)*

## Introduction

Depth is a loaded word in [machine learning](/wiki/machine_learning). In a [neural network](/wiki/neural_network), depth means the number of [layers](/wiki/layer) the input passes through on its way to becoming an output. In a [decision tree](/wiki/decision_tree), depth means the length of the longest path from the root node to any leaf. The two definitions live in different parts of the field, but they share a common idea: depth is the number of sequential decisions a model is allowed to make before committing to an answer.

The term carries weight because the phrase [deep learning](/wiki/deep_learning) is built directly on it. When Geoffrey Hinton and his collaborators began publishing on deep belief networks around 2006, the "deep" in the name referred to the architectural depth of the stack. That naming choice stuck. Today a model with three or more hidden layers is usually called deep, and a model with one hidden layer is usually called shallow.

This article covers both senses: depth in neural networks (how it is counted, why it matters, famous examples like [AlexNet](/wiki/alexnet), [VGG](/wiki/vgg), [ResNet](/wiki/resnet), and [GPT-3](/wiki/gpt-3)) and depth in decision trees and tree ensembles (including the `max_depth` hyperparameter and pruning).

## Depth of a neural network

A neural network is a stack of layers. Data enters at the [input layer](/wiki/input_layer), flows through a sequence of hidden layers, and exits at the [output layer](/wiki/output_layer). Each layer applies a transformation: a linear projection followed by a nonlinear [activation function](/wiki/activation_function). The depth of the network is the number of these transformations the data has to go through.

The input layer is usually not counted, because it does not actually transform anything. It just holds the input values. So a network described as having one input layer, three hidden layers, and one output layer is most often called a network of depth four, counting only the layers that do work.

Conventions vary. Some authors count only hidden layers; others count hidden plus output layers; others count weight layers (those with trainable parameters), excluding pooling and activation layers. Published deep learning papers usually count weight layers.

### Credit assignment path

A more formal version of depth is the credit assignment path (CAP). The CAP is the chain of transformations from input to output. For a feedforward network the CAP length is the number of hidden layers plus one. For a [recurrent neural network](/wiki/recurrent_neural_network), the CAP can be much larger because it includes unrolled time steps, so an RNN run on a 1,000 step sequence has an effective depth of more than 1,000 even if the cell itself has only a few layers.

Most researchers agree that deep learning involves CAP depth greater than two. There is no hard cutoff. In the 1980s a three layer network was considered ambitious. By the late 2010s a 100 layer network was routine.

### Why depth matters

The argument for depth is hierarchical feature learning. Early layers of a [convolutional neural network](/wiki/convolutional_neural_network) trained on images tend to detect simple things like edges and color blobs. Middle layers combine those into textures and shapes. Late layers combine those into object parts and whole objects. Each layer builds on the abstractions of the one below. You cannot get that hierarchy out of a single layer, no matter how wide it is.

A classical result called the universal approximation theorem says that even a network with one hidden layer can approximate any reasonable function, given enough neurons. So in principle, you do not need depth at all. In practice, you do. A shallow wide network that matches the accuracy of a deep network often needs an exponentially larger number of neurons. This is the depth efficiency or depth separation result.

### The cost of depth

More layers means more compute, more memory, and harder optimization. The most famous problem is the [vanishing gradient](/wiki/vanishing_gradient) problem. During [backpropagation](/wiki/backpropagation), the [gradient](/wiki/gradient) signal is repeatedly multiplied by the local derivatives of each layer's activation. If those derivatives are small, the product shrinks exponentially with depth, and the early layers receive almost no useful update. The opposite problem, exploding gradients, occurs when the derivatives are large.

The field has accumulated a toolkit for handling these issues. ReLU activations replaced sigmoid and tanh for most uses because they do not saturate on the positive side. [Batch normalization](/wiki/batch_normalization) and layer normalization keep activation distributions stable across layers. Residual or skip connections, introduced in ResNet, let the gradient flow around blocks instead of always passing through them. Weight initialization schemes such as He and Xavier initialization keep variances stable as you go deeper.

A second cost is overfitting. A very deep network has many parameters and can memorize the training set. This is countered with regularization, dropout, data augmentation, and larger datasets.

### Famous depths

The trajectory of depth in image classification shows how the field has scaled.

| Network | Year | Depth (weight layers) | Notes |
|---|---|---|---|
| LeNet-5 | 1998 | 7 | Yann LeCun's handwritten digit network. |
| AlexNet | 2012 | 8 | 5 convolutional plus 3 fully connected. Won ImageNet 2012. |
| VGG-16 | 2014 | 16 | 13 convolutional plus 3 fully connected. |
| VGG-19 | 2014 | 19 | 16 convolutional plus 3 fully connected. |
| GoogLeNet (Inception v1) | 2014 | 22 | Used inception modules to keep parameters down. |
| ResNet-50 | 2015 | 50 | First ResNet variant with bottleneck blocks. |
| ResNet-101 | 2015 | 101 | |
| ResNet-152 | 2015 | 152 | Won ImageNet 2015 with 3.57% top-5 error. |
| DenseNet-201 | 2016 | 201 | Each layer connects to all later layers. |

ResNet was the breakthrough that made very deep networks practical. The authors observed that stacking more layers actually made networks perform worse on training data, not just on test data. They called this the degradation problem. Their fix was to let each block learn a residual function on top of an identity shortcut. The skip connection lets the gradient take a shorter path back, which addresses both vanishing gradients and the optimization difficulty. A 152 layer ResNet was eight times deeper than VGG-19 yet had fewer parameters.

Language models pushed depth in a different direction. The original transformer paper from 2017 used 6 encoder and 6 decoder layers. By GPT-3 in 2020, the largest variant had 96 transformer decoder layers, 96 attention heads per layer, and 12,288 dimensional embeddings. PaLM, released in 2022, had 118 layers in its 540 billion parameter version. Modern frontier models have not always grown deeper in lockstep with parameter count; some have grown wider instead.

### Depth versus width

At fixed compute or parameter budget, designers face a tradeoff: should you add more layers, or more neurons per layer? Both are forms of capacity.

Depth tends to win for hierarchical tasks. Theoretical results on ReLU networks show that depth has a logarithmic relationship with the width needed to express certain function classes, meaning a deeper network can be exponentially smaller than a wide shallow network with comparable expressive power. Empirically, the deep learning revolution has been driven mostly by depth.

Width has its own advantages. Wider layers can process more features in parallel and are easier to train because the loss landscape is flatter. In practice, the best architectures balance the two, often pairing moderate depth with substantial width and many residual or attention connections to keep gradients well behaved.

## Depth of a decision tree

A [decision tree](/wiki/decision_tree) is a model that makes predictions by following a sequence of yes or no questions about the input features. The tree starts at a root node, each internal node splits the data based on one feature, and each leaf node holds a prediction. The depth of a decision tree is the length of the longest path from the root to any leaf, measured in edges. By that convention, a tree whose root is also a leaf has depth zero.

In a balanced tree, depth is roughly the logarithm of the number of leaves. In a heavily skewed tree, depth can grow much larger than that.

### Depth and overfitting

Depth controls how much capacity a tree has. A shallow tree can make only a few splits, so it underfits anything except very simple problems. A deep tree can make a unique split for nearly every training example, which lets it memorize the training set and generalize poorly. The relationship is direct: deeper trees have lower bias and higher variance. The right depth depends on the size and complexity of the dataset, and it is usually chosen by cross-validation.

In scikit-learn, the `max_depth` parameter of `DecisionTreeClassifier` and `DecisionTreeRegressor` is the standard way to limit depth. The documentation suggests starting with `max_depth=3`, visualizing the tree, and then increasing depth based on validation performance. The scikit-learn docs warn that the number of samples required to fill the tree doubles for each additional level, which is why unconstrained trees overfit so quickly on smaller datasets.

### Pre-pruning by depth

Limiting tree depth before or during training is a form of pre-pruning, also called early stopping. The idea is to halt growth based on a heuristic before the tree becomes too complex. Common pre-pruning parameters include:

| Parameter | Effect |
|---|---|
| `max_depth` | Hard cap on tree depth. |
| `min_samples_split` | Minimum samples required to split an internal node. |
| `min_samples_leaf` | Minimum samples required in a leaf. |
| `max_leaf_nodes` | Hard cap on the total number of leaves. |
| `min_impurity_decrease` | A split must improve impurity by at least this much. |

Pre-pruning is fast and easy to apply, but it can stop too early. A split that looks bad at the moment of evaluation might have led to two excellent splits one level below. This is sometimes called the horizon effect: the algorithm cannot see past the immediate split.

### Post-pruning by depth

Post-pruning grows the tree to its full depth first and then trims back branches that do not earn their keep. The most common technique in scikit-learn is minimal cost complexity pruning, parameterized by `ccp_alpha`. The algorithm produces a series of progressively simpler trees by repeatedly removing the subtree whose pruning gives the smallest increase in error per leaf removed. The right value of `ccp_alpha` is usually chosen by cross-validation on a separate split of the data.

Other post-pruning methods include reduced error pruning, where each internal node is tentatively replaced by a leaf labeled with the majority class, and the change is kept only if validation accuracy does not drop.

Post-pruning is more computationally expensive than pre-pruning but tends to produce better trees, because the algorithm sees the fully developed tree before deciding what to cut.

### Depth in tree ensembles

Most serious applications of trees today use ensembles. The two dominant families are bagging methods like [random forest](/wiki/random_forest) and boosting methods like [gradient boosting](/wiki/gradient_boosting), XGBoost, and LightGBM. Depth plays a different role in each.

In a random forest, the individual trees are usually grown deep, often with no depth limit. The variance reduction comes from averaging many decorrelated trees, so overfitting at the individual tree level is acceptable. The defaults in scikit-learn's `RandomForestClassifier` set `max_depth=None`.

In gradient boosting, the individual trees are usually shallow, often called weak learners. Depths of three to eight are typical. XGBoost defaults to `max_depth=6`. The shallowness keeps each tree from explaining too much variance on its own, which is the point of boosting: many small corrections beat one large overfit model.

## Explain like I'm 5

Imagine a stack of pancakes. Each pancake takes the stuff on top, mixes it, and passes it up. A short stack of two pancakes can only do simple mixing. A tall stack of fifty pancakes can do something fancy, like turning a photo of a cat into the word "cat," because each pancake handles a small step. That stack height is depth.

Now imagine twenty questions. The depth of a decision tree is how many questions you can ask before you have to guess. Too few and you cannot narrow it down. Too many and you ask weird questions that only work for the people you have already met.

## References

- Wikipedia, [Deep learning](https://en.wikipedia.org/wiki/Deep_learning)
- Wikipedia, [Residual neural network](https://en.wikipedia.org/wiki/Residual_neural_network)
- Wikipedia, [AlexNet](https://en.wikipedia.org/wiki/AlexNet)
- Wikipedia, [GPT-3](https://en.wikipedia.org/wiki/GPT-3)
- Wikipedia, [Decision tree pruning](https://en.wikipedia.org/wiki/Decision_tree_pruning)
- Simonyan, K. and Zisserman, A., [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/abs/1409.1556), 2014
- He, K. et al., Deep Residual Learning for Image Recognition, CVPR 2016
- Krizhevsky, A., Sutskever, I., Hinton, G., ImageNet Classification with Deep Convolutional Neural Networks, NeurIPS 2012
- Brown, T. et al., Language Models are Few-Shot Learners, NeurIPS 2020 (GPT-3)
- scikit-learn documentation, [Decision Trees](https://scikit-learn.org/stable/modules/tree.html)
- scikit-learn documentation, [DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)
- scikit-learn documentation, [Post pruning decision trees with cost complexity pruning](https://scikit-learn.org/stable/auto_examples/tree/plot_cost_complexity_pruning.html)