Neuron

See also: Machine learning terms, Neural network, Perceptron

A neuron (also called a node or unit) is the fundamental computational element of an artificial neural network: it takes one or more numeric inputs, multiplies each by a learned weight, sums them with a bias term, and passes the result through a nonlinear activation function to produce a single output. The idea dates to 1943, when Warren McCulloch and Walter Pitts published the first mathematical model of a neural unit, and it remains the building block of every modern deep learning system, from image classifiers to large language models with hundreds of billions of parameters.^[1] Inspired loosely by biological neurons in the nervous system, artificial neurons are organized into layers and connected through weighted links, forming the architecture of a neural network.

Historical Background

What was the McCulloch-Pitts neuron (1943)?

The concept of the artificial neuron dates back to 1943, when neurophysiologist Warren McCulloch and logician Walter Pitts published their landmark paper "A Logical Calculus of the Ideas Immanent in Nervous Activity" in the Bulletin of Mathematical Biophysics, volume 5, pages 115-133.^[1] Their model, now known as the McCulloch-Pitts (MCP) neuron, was the first mathematical model of a neural unit. It operated as a simple binary threshold device: the neuron received binary inputs, computed their sum, and fired (output 1) if the sum met or exceeded a fixed threshold, or remained silent (output 0) otherwise. Because of the "all-or-none" character of nervous activity, McCulloch and Pitts argued, neural events and the relations among them could be treated with propositional logic.^[1]

McCulloch and Pitts demonstrated that networks of these binary neurons could compute any Boolean logic function (AND, OR, NOT), meaning they were theoretically capable of universal computation. John von Neumann cited their work as a significant result that influenced early computer design. However, the MCP neuron had important limitations: all weights were fixed (there was no learning rule), and a single unit could not solve problems like XOR that require nonlinear separation.^[2]

When was the perceptron invented?

Frank Rosenblatt extended the MCP model by introducing the perceptron in 1958. The key innovation was that the perceptron had adjustable weights that could be trained from data using a learning algorithm. Rosenblatt's hardware implementation, the Mark I Perceptron, was built at the Cornell Aeronautical Laboratory under a U.S. Navy contract; it used 400 photocells arranged in a 20x20 grid as its "eye" and stored learned weights in motor-driven potentiometers.^[3] At a 1958 Navy press conference, Rosenblatt described the perceptron as "the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence."^[3]

Rosenblatt proved that the perceptron learning rule would converge to a correct solution for any linearly separable problem. Despite this advance, Marvin Minsky and Seymour Papert showed in their 1969 book Perceptrons that single-layer perceptrons could not learn non-linearly separable functions like XOR, which contributed to the first "AI winter."^[4]

How did modern neurons and backpropagation change the picture (1980s onward)?

In the 1980s, research on neural networks regained momentum with the development of the backpropagation algorithm for training multi-layer networks. This required neurons with smooth, differentiable activation functions (such as the sigmoid and tanh) rather than hard threshold steps. The transition to continuous activation functions enabled gradient-based optimization and opened the door to deep learning as it exists today.^[5]

Mathematical Model

An artificial neuron performs a straightforward computation that can be broken down into two stages: a linear transformation followed by a nonlinear activation.

The Neuron Equation

Given a neuron with n inputs, the output y is computed as:

y = f(w₁x₁ + w₂x₂ + ... + wₙxₙ + b)

Or in more compact notation:

y = f(w · x + b)

where:

Scroll sideways for more →

Step-by-Step Computation

1. Weighted sum: Each input xᵢ is multiplied by its corresponding weight wᵢ. These products are then summed together: z = Σ(wᵢ · xᵢ).
2. Add bias: The bias term b is added to the weighted sum: z = z + b. The bias allows the neuron to shift its activation threshold independently of the input values.
3. Apply activation function: The result z is passed through a nonlinear activation function f to produce the final output: y = f(z).

The weights and bias are the learnable parameters of the neuron. During training, they are adjusted iteratively using optimization algorithms (typically some variant of gradient descent) to minimize a loss function.

Components of a Neuron

Inputs

Inputs are the values fed into the neuron. In the first layer of a network, these come directly from the data (raw features such as pixel intensities or word embeddings). In deeper layers, the inputs are the outputs of neurons from the preceding layer.

Weights

Weights are scalar values that determine the strength and direction of the connection between an input and the neuron. A large positive weight means the corresponding input has a strong excitatory influence on the neuron's output. A large negative weight indicates an inhibitory influence. Weights are initialized (often randomly) before training begins and are updated through backpropagation.

Bias

The bias is an additional learnable parameter that allows the neuron to activate even when all inputs are zero. Geometrically, the bias shifts the decision boundary of the neuron away from the origin. Without a bias term, the neuron's decision hyperplane would always pass through the origin, limiting its flexibility.

Activation Function

The activation function introduces nonlinearity into the neuron's computation. Without it, any composition of neurons would still compute only a linear function, regardless of the number of layers. Common activation functions include:

Scroll sideways for more →

Single Neuron as a Linear Classifier

A single neuron with a step or sigmoid activation function can serve as a binary linear classifier. Its weighted sum w · x + b = 0 defines a hyperplane (a line in 2D, a plane in 3D) that separates the input space into two regions. Points on one side of the boundary are classified into one class, and points on the other side into the second class.

The bias shifts the position of this boundary, while the weights determine its orientation. This is precisely what the original perceptron does. However, a single neuron can only learn linearly separable patterns. Problems like XOR, where no straight line can separate the two classes, require multiple neurons organized across multiple layers.

How does an artificial neuron differ from a biological neuron?

Although artificial neurons draw inspiration from biology, the two are fundamentally different in complexity and mechanism. The human brain contains roughly 86 billion neurons, each forming on the order of 7,000 synaptic connections on average, yet runs on about 20 watts of power, a level of efficiency that today's GPU-based networks cannot approach.^[6]

Scroll sideways for more →

It is worth noting that artificial neurons are simplified "caricature models" of their biological counterparts. Real neurons exhibit complex behaviors such as dendritic computation, diverse neurotransmitter dynamics, and temporal coding that artificial neurons do not capture.^[7]

Types of Neurons by Role

Neurons are classified by their position and function within the network:

Scroll sideways for more →

Neurons in Different Layer Types

The behavior of a neuron depends heavily on the type of layer it belongs to:

Dense (Fully Connected) Layers

In a dense layer, every neuron is connected to every neuron in the previous layer. Each neuron computes the full weighted sum over all its inputs. Dense layers are the most general type and are commonly used in the final classification stages of a network.

Convolutional Layers

In a convolutional neural network (CNN), neurons are arranged in three dimensions (height, width, depth) and each neuron is connected to only a small local region (receptive field) of the previous layer. All neurons in a feature map share the same weights (a filter or kernel), which dramatically reduces the number of parameters and allows the network to detect spatial patterns like edges, textures, and objects regardless of their position in the input.

Recurrent Layers

In a recurrent neural network (RNN), neurons have connections that loop back on themselves, giving the network a form of memory. At each time step, a recurrent neuron receives both a new input and its own previous output (or hidden state), allowing it to process sequential data such as text, speech, or time series. Variants like LSTM and GRU add gating mechanisms that control how information flows through recurrent neurons over time.

Attention Layers

In transformer architectures, the attention mechanism computes dynamic, data-dependent weighted sums rather than using fixed connections. While the underlying linear projections still use standard neuron-like units, the attention mechanism allows each position in a sequence to attend to all other positions, breaking free from the fixed connectivity patterns of dense and convolutional layers.

Number of Neurons and Model Capacity

The number of neurons in a neural network is directly tied to the model's capacity, which describes the range of functions the model can learn.

• Width (number of neurons per layer): Increasing width allows a layer to represent more features simultaneously. In theory, a single hidden layer with enough neurons can approximate any continuous function (the Universal Approximation Theorem). In practice, very wide single-layer networks are difficult to train effectively.
• Depth (number of layers): Increasing depth allows the network to learn hierarchical, compositional features. Research has shown that the number of distinct linear regions a network can represent grows exponentially with depth but only polynomially with width, making depth a more efficient way to increase model complexity.^[8]
• Overfitting vs. underfitting: A model with too few neurons may underfit (fail to capture the patterns in the data). A model with too many neurons may overfit (memorize training data rather than learning general patterns). Techniques like dropout, regularization, and early stopping help manage this tradeoff.

Modern large language models contain billions of parameters distributed across millions of neuron-like units organized in deep architectures, giving them the capacity to model extremely complex language patterns.

Dead Neurons and the Dying ReLU Problem

A dead neuron is one that always outputs zero regardless of its input, effectively contributing nothing to the network's computation. This phenomenon is most commonly associated with the ReLU activation function.

How It Happens

ReLU outputs zero for any negative input: f(z) = max(0, z). If, during training, the weights and bias of a neuron shift such that its pre-activation value z becomes negative for all training examples, the neuron will consistently output zero. Because the gradient of ReLU is also zero for negative inputs, the neuron receives no gradient signal during backpropagation, meaning its weights can never be updated. The neuron is effectively "dead."^[9]

Causes

• Large learning rates that cause weights to overshoot into regions where the neuron's input is always negative.
• Poor weight initialization that places neurons in permanently inactive regions from the start.
• Unfortunate data distributions that cause certain neurons to receive predominantly negative inputs.

Solutions

Scroll sideways for more →

If more than 40% of neurons in a layer become dead, it can severely impair the network's ability to learn.

What do individual neurons learn? (Interpretability)

Understanding what individual neurons learn is a central question in neural network interpretability.

Feature Visualization

Researchers use optimization techniques to generate synthetic inputs that maximally activate a specific neuron, revealing what "feature" or pattern the neuron has learned to detect. In image networks like Inception and ResNet, these visualizations show that early-layer neurons respond to simple patterns (edges, colors), middle-layer neurons detect textures and parts (eyes, wheels), and late-layer neurons respond to complex objects or scenes.^[10]

Dataset Examples

A complementary approach involves feeding a large dataset through the network and identifying which real inputs produce the highest activation for a given neuron. This provides intuitive evidence of what each neuron "looks for."

What is a polysemantic neuron?

Many neurons turn out to be polysemantic, meaning they respond to multiple unrelated concepts. A well-documented example is neuron 4e:55 in the InceptionV1 vision model, which responds to cat faces, cat legs, and the fronts of cars: feature visualization shows it is not detecting some subtle shared feature but genuinely responding to all three.^[10] In language models, a single neuron might respond to academic citations, dialogue, HTTP requests, and Korean text simultaneously. This makes interpreting individual neurons difficult.

What is the superposition hypothesis?

The superposition hypothesis, articulated by researchers at Anthropic in their 2022 paper "Toy Models of Superposition," offers an explanation for polysemantic neurons. The paper investigates "how and when models represent more features than they have dimensions," a phenomenon the authors call superposition.^[11] The core idea is that neural networks need to represent more features than they have neurons, so they exploit the geometry of high-dimensional spaces to encode multiple features as overlapping directions in activation space. Each neuron participates in encoding many features simultaneously, and each feature is distributed across many neurons. The paper shows that superposition lets a model store extra features at the cost of "interference" that requires nonlinear filtering, and that features organize into geometric structures such as digons, triangles, and pentagons.^[11]

This insight has driven the development of techniques like sparse autoencoders, which decompose polysemantic neuron activations into more interpretable "monosemantic" features. Anthropic's 2023 follow-up, "Towards Monosemanticity," used a sparse autoencoder to extract thousands of interpretable features from a one-layer transformer.^[12] Understanding superposition is considered one of the most important open problems in AI interpretability and safety research.

Explain Like I'm 5 (ELI5)

Imagine a neuron in a neural network is like a tiny worker in a huge factory. The worker gets messages from many other workers. Some messages are important (they have a big "weight"), and some are less important (small "weight"). The worker adds up all the messages, paying more attention to the important ones. Then the worker checks: "Is this total big enough for me to care?" If it is, the worker gets excited and passes along a signal to the next group of workers. If not, the worker stays quiet.

When you stack thousands of these workers in rows (layers), the first row notices simple things (like edges in a picture), the next row combines those simple things into medium things (like eyes or wheels), and the deeper rows recognize whole objects (like a cat or a car). The factory learns by adjusting how much each worker pays attention to each message, trying to get the right answer more often.

References

1. McCulloch, W. S., & Pitts, W. (1943). "A Logical Calculus of the Ideas Immanent in Nervous Activity." *Bulletin of Mathematical Biophysics*, 5(4), 115-133. https://link.springer.com/article/10.1007/BF02478259 ↩
2. "Artificial neuron." *Wikipedia*. https://en.wikipedia.org/wiki/Artificial_neuron ↩
3. "Perceptron." *Wikipedia*. https://en.wikipedia.org/wiki/Perceptron ↩
4. Minsky, M., & Papert, S. (1969). *Perceptrons: An Introduction to Computational Geometry*. MIT Press. ↩
5. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." *Nature*, 323(6088), 533-536. ↩
6. "Billions of neurons, trillions of synapses." *UCLA Brain Research Institute*. https://bri.ucla.edu/brain-fact/billions-of-neurons-trillions-of-synapses/ ↩
7. "Biological Neurons vs Artificial Neural Networks." *Sophos Blog*. https://www.sophos.com/en-us/blog/man-vs-machine-comparing-artificial-and-biological-neural-networks ↩
8. Montufar, G. F., Pascanu, R., Cho, K., & Bengio, Y. (2014). "On the Number of Linear Regions of Deep Neural Networks." *Advances in Neural Information Processing Systems (NeurIPS)*. ↩
9. Lu, L., Shin, Y., Su, Y., & Karniadakis, G. E. (2019). "Dying ReLU and Initialization: Theory and Numerical Examples." *arXiv:1903.06733*. ↩
10. Olah, C., Mordvintsev, A., & Schubert, L. (2017). "Feature Visualization." *Distill*. https://distill.pub/2017/feature-visualization/ ↩
11. Elhage, N., et al. (2022). "Toy Models of Superposition." *Transformer Circuits Thread*. https://transformer-circuits.pub/2022/toy_model/index.html ↩
12. Bricken, T., et al. (2023). "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." *Transformer Circuits Thread*. https://transformer-circuits.pub/2023/monosemantic-features ↩