Activation Function
An activation function is a nonlinear mathematical function applied to the output of each neuron in an artificial neural network, and it is what gives the network the ability to learn complex
Neural Networks
Explore Neural Networks through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of Neural Networks.
All articles (116)Deep Learning (81)Machine Learning (58)Computer Vision (17)Model Architecture (15)Interpretability (6)AI History (5)Artificial Intelligence (4)Generative AI (4)Natural Language Processing (4)Transformer Models (4)AI Hardware (3)AI Research (3)Mathematics (3)Google DeepMind (2)Information Retrieval (2)Open Source AI (2)People (2)AI Ethics (1)AI Infrastructure (1)AI for Science (1)Developer Tools (1)Google (1)Large Language Models (1)MLOps (1)Model Evaluation (1)Programming Languages (1)Statistics (1)Training & Optimization (1)
Showing 1-60 of 81 articles
Attention
Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them.
Attention sink
An attention sink is an empirical phenomenon in Transformer language models in which a large fraction of each attention head's weight concentrates on a few tokens at the very start of the sequence, most often…
Backpropagation
Backpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation.
Batch Normalization
Batch normalization (BatchNorm or BN) is a normalization method used inside some neural networks.
Bayesian Neural Network
A Bayesian neural network (BNN) is a neural network in which the weights and biases are represented as probability distributions rather than fixed point estimates.
Calibration Layer
A calibration layer is a post-prediction adjustment appended to a trained machine learning model that rescales its raw output scores or predicted probabilities so they better reflect the true likelihood of…
Co-Adaptation
Co-adaptation in neural networks refers to a phenomenon in which different hidden units develop highly correlated behavior, becoming excessively dependent on one another rather than learning independent
Connectionism
Connectionism is an approach in cognitive science and artificial intelligence that explains mental phenomena as the collective behavior of large networks of simple, interconnected processing units.
Continual learning
Continual learning, also called lifelong learning or incremental learning, is a machine learning paradigm in which a model learns from a stream of tasks or data distributions over time
ConvNeXt
ConvNeXt is a family of pure convolutional neural network (CNN) models that match or beat Vision Transformers on standard vision benchmarks
Convolutional Neural Network
A convolutional neural network (CNN, or ConvNet) is a neural network that uses convolution-like linear operators in at least some layers.
Decoder
A decoder is the component of a neural network that turns an internal, compressed, or abstract representation into a desired output, such as a translated sentence, a generated image, a reconstructed input, or…
Deep Learning
Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to automatically learn representations of data at multiple levels of abstraction
Deep Neural Network
A deep neural network (DNN) is an artificial neural network with multiple hidden layers of artificial neurons stacked between its input and output layers
DeepNorm / DeepNet
DeepNorm is a normalization and weight initialization scheme for Transformer networks that makes the training of very deep models stable.
DeepSeek Sparse Attention (DSA)
DeepSeek Sparse Attention (DSA) is a trainable, fine-grained sparse attention mechanism introduced by the Chinese AI company DeepSeek in its experimental model DeepSeek-V3.2-Exp, released on September 29, 2025.
Dense Layer
A dense layer, also called a fully connected (FC) layer, linear layer, or affine layer, is a layer in an artificial neural network where every input neuron is connected to every output neuron.
DenseNet
DenseNet (Densely Connected Convolutional Networks) is a convolutional neural network architecture that connects every layer to every other layer in a feed-forward fashion
EfficientNet
EfficientNet is a family of convolutional neural network architectures and a model-scaling method that uniformly scales network depth, width, and input resolution with a single compound coefficient, developed…
Epoch
An epoch in machine learning is one complete pass through the entire training dataset, during which every example is presented to the model exactly once to compute gradients and update parameters before the…
Expert Choice routing
Expert Choice routing (often abbreviated EC) is a routing method for mixture of experts (MoE) layers in neural networks, introduced by researchers at Google in 2022.
Exploding Gradient Problem
The exploding gradient problem is a training failure in deep neural networks where the gradients of the loss function grow exponentially large as they propagate backward through layers during backpropagation…
Feedforward Neural Network (FFN)
A feedforward neural network (FFN), also called a multilayer perceptron (MLP) when it has multiple layers, is a type of artificial neural network in which information flows in one direction only, from the…
Full Softmax
Full softmax (also called the standard softmax or exact softmax) is the softmax computation that calculates a probability for every possible output class in a classification model
Fully Connected Layer
A fully connected layer (also called a dense layer or linear layer) is a layer in an artificial neural network in which every input value connects to every output value through a learned weight
GELU (Gaussian Error Linear Unit)
The Gaussian Error Linear Unit (GELU) is a smooth, non-monotonic activation function defined as GELU(x) = x · Φ(x), where Φ(x) is the cumulative distribution function of the standard normal distribution.
Gated DeltaNet
Gated DeltaNet is a linear attention sequence model that augments the delta rule, a key-value error-correction update used in the earlier DeltaNet architecture
Generative adversarial network
Generative adversarial networks (GANs) are a family of generative models trained through competition between two learned functions.
Graph Neural Network
A Graph Neural Network (GNN) is a class of neural_network that operates directly on graph-structured data, meaning data made of nodes (entities) connected by edges (relationships), by repeatedly passing…
H-Net (dynamic chunking)
H-Net, short for Hierarchical Network, is a tokenizer-free neural sequence model that learns to segment raw bytes into content-adaptive "chunks" as part of ordinary end-to-end training, rather than relying on…
Hidden Layer
A hidden layer is a layer of artificial neurons in a neural network that sits between the input layer and the output layer
Hopfield Network
A Hopfield network is a form of recurrent neural network that stores information as stable patterns in a system of interconnected binary units and retrieves it by content rather than by address.
Hymba
Hymba is a hybrid-head neural-network architecture for small language models introduced by NVIDIA researchers in November 2024.
Inception (deep learning)
Inception is a family of convolutional neural network (CNN) architectures developed by researchers at Google, first introduced in 2014.
Input Layer
The input layer is the first layer of a neural network: it receives the raw feature vector for each data sample and passes those values forward to the next layer, performing no learned computation of its own.
John Hopfield
John Joseph Hopfield (born July 15, 1933) is an American physicist whose work spans condensed-matter physics, molecular biology, and the theory of neural computation.
Jürgen Schmidhuber
Jürgen Schmidhuber (anglicized Jurgen Schmidhuber, born 17 January 1963) is a German computer scientist best known as a co-inventor of long short-term memory (LSTM), the recurrent neural network architecture…
Kolmogorov-Arnold Network
A Kolmogorov-Arnold Network (KAN) is a type of neural network architecture proposed as an alternative to the traditional Multi-Layer Perceptron (MLP).
Layer
A layer is the fundamental building block of a neural network: an organized group of neurons (also called nodes or units) that together apply one mathematical transformation to their input and pass the result…
LeNet
LeNet is the pioneering family of convolutional neural networks developed by Yann LeCun and collaborators at AT&T Bell Labs between roughly 1988 and 1998 to read handwritten characters
Lightning Attention
Lightning Attention is an IO-aware (input/output aware) implementation of linear attention that lets the method reach its theoretical linear-time complexity in practice
Logits
In machine learning and statistics, logits refer to the raw, unnormalized scores output by a model before they are converted into probabilities.
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences.
Mixture of Block Attention (MoBA)
Mixture of Block Attention (MoBA) is a trainable block-sparse attention mechanism introduced in February 2025 by researchers at Moonshot AI, the company that builds the Kimi family of large language models
Mixture of Experts (MoE)
A mixture of experts (MoE) is a machine-learning architecture that contains multiple component functions called experts and a gating or routing mechanism that determines how their outputs contribute to a…
Mixture-of-Recursions (MoR)
Mixture-of-Recursions (MoR) is a Transformer architecture, introduced in 2025, that unifies two previously separate strategies for building efficient language models: parameter sharing through recursion, and…
MobileNet
MobileNet is a family of efficient convolutional neural network (CNN) architectures developed by Google for mobile and edge AI applications.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Multilayer Perceptron
A multilayer perceptron (MLP) is a feedforward neural network built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer
Native Sparse Attention (NSA)
Native Sparse Attention (NSA) is a hardware-aligned, natively trainable sparse attention mechanism introduced in February 2025 by DeepSeek, in collaboration with researchers at Peking University and the…
NeRF
Neural Radiance Fields (NeRF) is a method for synthesizing photorealistic novel views of a 3D scene by encoding the scene as a continuous 5D function (3D position plus 2D viewing direction) inside a single…
Neural Network
A neural network (also called an artificial neural network or ANN) is a computational model, loosely inspired by the networks of biological neurons in animal brains
Neural Tangent Kernel
The neural tangent kernel (NTK) is a kernel function built from the parameter gradients of a neural network.
Neural architecture search
Neural architecture search (NAS) is a technique for automating the design of neural network architectures.
Output Layer
The output layer is the final layer of a neural network: it takes the features computed by the hidden layers and converts them into the model's prediction, with its size and activation function fixed by the…
PEER (Parameter Efficient Expert Retrieval / Mixture of a Million Experts)
PEER, short for Parameter Efficient Expert Retrieval, is a neural network layer for Transformer models that replaces the dense feed-forward block with a sparse mixture of experts drawn from an extremely large…
Position Interpolation (PI)
Position Interpolation (PI) is a method for extending the context window of a pretrained large language model that uses rotary position embedding (RoPE).
RNN
RNN is the standard abbreviation for recurrent neural network, a class of artificial neural network in which connections between units form cycles