# Multilayer Perceptron

> Source: https://aiwiki.ai/wiki/multilayer_perceptron
> Updated: 2026-07-24
> Fact-checked: 2026-07-24
> Categories: AI History, Deep Learning, Model Architecture, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Multilayer Perceptron." aiwiki.ai, 24 Jul 2026. https://aiwiki.ai/wiki/multilayer_perceptron
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

A multilayer perceptron (MLP) is a feedforward [neural network](https://aiwiki.ai/wiki/neural_network) built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer, with a nonlinear [activation function](https://aiwiki.ai/wiki/activation_function) applied at each hidden unit. Data flows in one direction, from input to output, with no recurrent connections. Each layer computes a weighted sum of the previous layer's outputs, adds a bias, and passes the result through the activation; the weights are learned from examples, almost always by [gradient descent](https://aiwiki.ai/wiki/gradient_descent) with gradients computed via [backpropagation](https://aiwiki.ai/wiki/backpropagation) [4].

The MLP is the oldest neural architecture still in everyday use. It descends from Frank Rosenblatt's [perceptron](https://aiwiki.ai/wiki/perceptron) of 1958 [1], survived the criticism of Marvin Minsky and Seymour Papert's 1969 book Perceptrons [3], and became trainable in practice once backpropagation was popularized in 1986 [4]. Universal approximation theorems proved between 1989 and 1993 established that even a single hidden layer can approximate any continuous function on a bounded domain to arbitrary accuracy, given enough units [6][7][9].

Although convolutional and attention-based architectures took over perception and language tasks, the MLP never left. Every [transformer](https://aiwiki.ai/wiki/transformers) block contains one: the position-wise feed-forward network that follows attention is a two-layer MLP, and these layers hold a large share of the parameters in modern [large language models](https://aiwiki.ai/wiki/large_language_model) [11][16]. MLPs also serve as classification heads, projection layers, and the expert modules in [mixture of experts](https://aiwiki.ai/wiki/mixture_of_experts) models [18], and periodically return as complete architectures in their own right, as in MLP-Mixer (2021) [19]. Since 2024, Kolmogorov-Arnold networks have been studied as a proposed alternative [20].

## From the perceptron to Minsky and Papert

Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory in Buffalo, New York, published "The perceptron: a probabilistic model for information storage and organization in the brain" in Psychological Review in 1958 [1]. The perceptron was a brain-inspired model that learned to classify patterns by adjusting connection weights. Rosenblatt and his colleagues also built it in hardware: the Mark I Perceptron, assembled and tested during 1959 and demonstrated publicly in June 1960, took input from 400 photocells arranged in a 20x20 grid and fed them through a layer of association units to eight output units [2].

The trainable part of Rosenblatt's machine was a single layer of adjustable weights, and that is where the trouble lay. A single layer can only draw linear decision boundaries, so there are simple functions it can never represent, the exclusive-or (XOR) of two inputs being the standard example. [Marvin Minsky](https://aiwiki.ai/wiki/marvin_minsky) and Seymour Papert made this precise in Perceptrons: An Introduction to Computational Geometry (MIT Press, 1969), which proved a set of limitations on single-layer perceptrons, including the XOR case, lower bounds for computing parity, and the rapidly growing resource requirements of deciding whether a figure is connected [3].

Minsky and Papert knew that networks with more layers escape these particular proofs, but they were openly skeptical that such extensions would lead anywhere, conjecturing that the multilayer direction was "sterile" [3]. The book is widely credited with helping redirect research money toward symbolic AI in the 1970s, a period now folded into the history of the first [AI winter](https://aiwiki.ai/wiki/ai_winter), though the authors maintained that perceptron research declined because of the technique's own limits rather than their critique [3]. An expanded edition with a new prologue and epilogue appeared in 1988, responding to the neural network revival then underway [3].

## Backpropagation and the return of hidden layers

Adding hidden layers solves the representation problem and creates a new one: how to assign credit or blame to weights buried inside the network. Rosenblatt had used the phrase "back-propagating error correction" as early as 1962 without knowing how to implement it [22]. The pieces accumulated over the following decades: Seppo Linnainmaa introduced the reverse mode of automatic differentiation, the algorithmic core of backpropagation, in his 1970 master's thesis [22], and Paul Werbos described training neural networks this way in his 1974 Harvard dissertation, "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences" [5].

What made the method famous was the 1986 Nature paper "Learning representations by back-propagating errors" by David Rumelhart, [Geoffrey Hinton](https://aiwiki.ai/wiki/geoffrey_hinton), and Ronald Williams [4]. The paper showed that repeatedly adjusting weights to reduce output error causes hidden units to develop useful internal feature detectors, and argued that this ability to create new features is exactly what separates the method from the perceptron convergence procedure of the 1960s [4]. The result set off the connectionist revival of the late 1980s, with the MLP trained by backpropagation as its workhorse model.

## Architecture and training

An MLP with one hidden layer computes, in sequence: a linear map from inputs to hidden pre-activations, an elementwise nonlinearity, and a second linear map to the outputs. Deeper MLPs stack more hidden layers. Because every unit in one layer connects to every unit in the next, the layers are called fully connected (or dense), which distinguishes MLPs from [convolutional neural networks](https://aiwiki.ai/wiki/convolutional_neural_network), whose connections are local and weight-shared. The nonlinearity is essential: a stack of purely linear layers collapses into a single linear map, no more expressive than the perceptron.

The choice of activation function has changed several times over the architecture's history.

| Activation | Introduced or popularized | Notes |
| --- | --- | --- |
| Step function | Rosenblatt, 1958 [1] | Not differentiable, so unusable with backpropagation |
| Sigmoid / tanh | Standard in the 1980s-1990s backpropagation era [4] | Smooth, but saturates; deep stacks suffer vanishing gradients |
| [ReLU](https://aiwiki.ai/wiki/relu) | Nair and Hinton, 2010 [12]; Glorot, Bordes and Bengio, 2011 [13] | max(0, x); enabled training deep supervised networks without pretraining |
| [GELU](https://aiwiki.ai/wiki/gelu) | Hendrycks and Gimpel, 2016 [14] | x * Phi(x), a smooth ReLU alternative common in transformer FFNs [15] |
| [SwiGLU](https://aiwiki.ai/wiki/swiglu) and other GLU variants | Shazeer, 2020 [15] | Gated products of two linear projections; adopted by Llama-class models [16] |

Nair and Hinton introduced rectified linear units in restricted Boltzmann machines in 2010 [12], and Glorot, Bordes, and Bengio showed the next year that rectifier networks match or beat tanh networks despite the kink at zero, producing sparse activations and training well on labeled data without any unsupervised pretraining [13].

Training is ordinary [supervised learning](https://aiwiki.ai/wiki/supervised_learning): pick a loss function, compute its gradient with respect to every weight by backpropagation, and update the weights by gradient descent or a variant such as stochastic gradient descent with minibatches. Regularization techniques developed for deep networks generally, including [dropout](https://aiwiki.ai/wiki/dropout) and normalization layers, apply to MLPs as well.

## Universal approximation

The theoretical case for hidden layers arrived in 1989. George Cybenko proved in "Approximation by superpositions of a sigmoidal function" (Mathematics of Control, Signals, and Systems, vol. 2, pp. 303-314) that a feedforward network with a single hidden layer of sigmoidal units can uniformly approximate any continuous function on the unit hypercube to any desired accuracy [6]. The same year, Kurt Hornik, Maxwell Stinchcombe, and Halbert White reached a similar conclusion for a much broader class of activation functions in Neural Networks [7], and in 1991 Hornik showed that the power comes from the multilayer feedforward architecture itself rather than from any specific activation, so long as the activation is nonconstant and bounded [8]. Leshno and colleagues completed the picture in 1993: a locally bounded activation yields universal approximation exactly when it is not a polynomial [9][10].

These results, collectively known as the [universal approximation theorem](https://aiwiki.ai/wiki/universal_approximation_theorem), are existence statements. They guarantee that suitable weights exist, but say nothing about how many hidden units are required, whether gradient-based training will find those weights, or how the network will behave on data it was not fit to. The gap between what an MLP can represent and what training reliably finds has been a running theme of [deep learning](https://aiwiki.ai/wiki/deep_learning) theory ever since.

## MLPs inside transformers

The transformer architecture of Vaswani et al. (2017) alternates two sublayers: multi-head attention, and a position-wise feed-forward network that is applied identically and independently at every token position [11]. That feed-forward network is a two-layer MLP: in the paper's formulation, FFN(x) = max(0, xW1 + b1)W2 + b2, a linear expansion followed by a ReLU and a linear projection back down. In the base model the expansion goes from a model width of 512 up to an inner width of 2,048, a factor of four [11].

Later language models kept the structure and swapped the details. Shazeer proposed replacing the ReLU or GELU in the FFN with gated linear unit variants in 2020 [15], and Meta's LLaMA models adopted his SwiGLU activation, shrinking the inner dimension to two thirds of 4d to keep the parameter count comparable [16]. Because the FFN expands and contracts at every layer, these MLP blocks account for a large fraction of a transformer's weights, which is why mixture of experts designs target them specifically: the Switch Transformer replaces each dense FFN with a pool of expert FFNs and routes every token to one of them, a trick that let Google scale models past a trillion parameters in 2021 without increasing per-token compute [18].

Interpretability research suggests these MLP blocks are where much of a model's stored knowledge lives. Geva and colleagues argued at EMNLP 2021 that transformer feed-forward layers operate as key-value memories: the first layer's weight rows act as keys that respond to textual patterns in the training data, the second layer's columns act as values that push probability toward particular output tokens, and lower layers respond to shallow patterns while upper layers respond to more semantic ones [17].

## All-MLP vision architectures

In May 2021, Tolstikhin and colleagues asked whether vision models need convolution or attention at all. Their answer, MLP-Mixer, is an architecture built exclusively from MLPs: images are cut into patches, and the network alternates "channel-mixing" MLPs applied independently to each patch with "token-mixing" MLPs applied across patches to move spatial information around [19]. Trained on large datasets or with modern regularization, Mixer scored competitively with contemporary CNNs and [vision transformers](https://aiwiki.ai/wiki/vision_transformer) on image classification benchmarks, and the paper (published at NeurIPS 2021) concluded that neither convolutions nor attention are necessary for good performance, even if both are sufficient [19]. Several related all-MLP vision models followed, though attention-based architectures remained dominant in practice.

## Kolmogorov-Arnold networks

In April 2024, Ziming Liu and collaborators proposed Kolmogorov-Arnold networks (KANs) as an alternative to MLPs, drawing on the Kolmogorov-Arnold representation theorem rather than universal approximation [20]. Where an MLP has fixed activation functions on its units and learnable linear weights on its edges, a [KAN](https://aiwiki.ai/wiki/kolmogorov_arnold_network) has learnable activation functions on its edges, each parametrized as a spline, and no linear weight matrices at all. The paper reported that much smaller KANs matched or beat larger MLPs on data-fitting and partial differential equation benchmarks, claimed faster empirical scaling, and emphasized interpretability, showing examples where the learned splines helped researchers rediscover mathematical and physical laws [20]. The work was accepted at ICLR 2025 [20].

Independent follow-up work complicated the picture. A controlled comparison by Yu, Yu, and Wang, holding parameter counts and FLOPs equal, found that standard MLPs outperformed KANs across machine learning, computer vision, audio, and natural language tasks, with KANs ahead only on symbolic formula representation; the authors traced that remaining advantage to the B-spline activation, which also helped MLPs when transplanted into them, and reported that KANs suffered worse catastrophic forgetting than MLPs in class-incremental continual learning, contrary to a claim in the original paper [21]. Whether KANs or other learned-activation designs find a durable niche remains an open question.

## Milestones

| Year | Event |
| --- | --- |
| 1958 | Rosenblatt publishes the perceptron in Psychological Review [1] |
| 1960 | Mark I Perceptron hardware demonstrated publicly [2] |
| 1969 | Minsky and Papert publish Perceptrons, proving limits of single-layer machines [3] |
| 1970 | Linnainmaa introduces reverse-mode automatic differentiation in his master's thesis [22] |
| 1974 | Werbos's Harvard dissertation describes training networks by backpropagating errors [5] |
| 1986 | Rumelhart, Hinton, and Williams popularize backpropagation in Nature [4] |
| 1989 | Cybenko, and Hornik, Stinchcombe, and White, prove universal approximation [6][7] |
| 1993 | Leshno et al. show nonpolynomial activation is necessary and sufficient [9][10] |
| 2010-2011 | ReLU shown to ease deep network training [12][13] |
| 2017 | Transformer places a two-layer MLP (the FFN) in every block [11] |
| 2021 | MLP-Mixer revives all-MLP vision models; Switch Transformer scales expert FFNs to a trillion parameters [18][19] |
| 2024 | Kolmogorov-Arnold networks proposed as an MLP alternative [20] |

## See also

- [Perceptron](https://aiwiki.ai/wiki/perceptron)
- [Backpropagation](https://aiwiki.ai/wiki/backpropagation)
- [Universal approximation theorem](https://aiwiki.ai/wiki/universal_approximation_theorem)
- [Feedforward neural network (FFN)](https://aiwiki.ai/wiki/feedforward_neural_network_ffn)
- [Activation function](https://aiwiki.ai/wiki/activation_function)
- [Kolmogorov-Arnold network](https://aiwiki.ai/wiki/kolmogorov_arnold_network)

## References

1. Rosenblatt, F. "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain." Psychological Review, vol. 65, no. 6, 1958, pp. 386-408. https://homepages.math.uic.edu/~lreyzin/papers/rosenblatt58.pdf
2. Wikipedia. "Perceptron." https://en.wikipedia.org/wiki/Perceptron
3. Wikipedia. "Perceptrons (book)" (Minsky, M. and Papert, S. Perceptrons: An Introduction to Computational Geometry. MIT Press, 1969; expanded edition 1988). https://en.wikipedia.org/wiki/Perceptrons_(book)
4. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. "Learning representations by back-propagating errors." Nature, vol. 323, pp. 533-536, October 9, 1986. https://www.nature.com/articles/323533a0
5. Wikipedia. "Paul Werbos." https://en.wikipedia.org/wiki/Paul_Werbos
6. Cybenko, G. "Approximation by superpositions of a sigmoidal function." Mathematics of Control, Signals, and Systems, vol. 2, pp. 303-314, 1989. https://link.springer.com/article/10.1007/BF02551274
7. Hornik, K., Stinchcombe, M., and White, H. "Multilayer feedforward networks are universal approximators." Neural Networks, vol. 2, no. 5, pp. 359-366, 1989. https://doi.org/10.1016/0893-6080(89)90020-8
8. Hornik, K. "Approximation capabilities of multilayer feedforward networks." Neural Networks, vol. 4, no. 2, pp. 251-257, 1991. https://doi.org/10.1016/0893-6080(91)90009-T
9. Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S. "Multilayer feedforward networks with a nonpolynomial activation function can approximate any function." Neural Networks, vol. 6, no. 6, pp. 861-867, 1993. https://doi.org/10.1016/S0893-6080(05)80131-5
10. Wikipedia. "Universal approximation theorem." https://en.wikipedia.org/wiki/Universal_approximation_theorem
11. Vaswani, A., et al. "Attention Is All You Need." arXiv:1706.03762, June 2017. https://arxiv.org/abs/1706.03762
12. Nair, V. and Hinton, G. E. "Rectified Linear Units Improve Restricted Boltzmann Machines." Proceedings of the 27th International Conference on Machine Learning (ICML), 2010. https://icml.cc/Conferences/2010/papers/432.pdf
13. Glorot, X., Bordes, A., and Bengio, Y. "Deep Sparse Rectifier Neural Networks." Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), 2011. https://proceedings.mlr.press/v15/glorot11a.html
14. Hendrycks, D. and Gimpel, K. "Gaussian Error Linear Units (GELUs)." arXiv:1606.08415, June 2016. https://arxiv.org/abs/1606.08415
15. Shazeer, N. "GLU Variants Improve Transformer." arXiv:2002.05202, February 2020. https://arxiv.org/abs/2002.05202
16. Touvron, H., et al. "LLaMA: Open and Efficient Foundation Language Models." arXiv:2302.13971, February 2023. https://arxiv.org/abs/2302.13971
17. Geva, M., Schuster, R., Berant, J., and Levy, O. "Transformer Feed-Forward Layers Are Key-Value Memories." EMNLP 2021; arXiv:2012.14913. https://arxiv.org/abs/2012.14913
18. Fedus, W., Zoph, B., and Shazeer, N. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." arXiv:2101.03961, January 2021. https://arxiv.org/abs/2101.03961
19. Tolstikhin, I., et al. "MLP-Mixer: An all-MLP Architecture for Vision." NeurIPS 2021; arXiv:2105.01601. https://proceedings.neurips.cc/paper/2021/hash/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Abstract.html
20. Liu, Z., et al. "KAN: Kolmogorov-Arnold Networks." arXiv:2404.19756, April 2024 (ICLR 2025). https://arxiv.org/abs/2404.19756
21. Yu, R., Yu, W., and Wang, X. "KAN or MLP: A Fairer Comparison." arXiv:2407.16674, July 2024. https://arxiv.org/abs/2407.16674
22. Wikipedia. "Backpropagation" (history and precursors). https://en.wikipedia.org/wiki/Backpropagation