Activation Function
An activation function is a nonlinear mathematical function applied to the output of each neuron in an artificial neural network, and it is what gives the network the ability to learn complex
Neural Networks
Explore Neural Networks through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (28)
Articles that also belong to these categories. Counts cover all of Neural Networks.
All articles (116)Deep Learning (81)Machine Learning (58)Computer Vision (17)Model Architecture (15)Interpretability (6)AI History (5)Artificial Intelligence (4)Generative AI (4)Natural Language Processing (4)Transformer Models (4)AI Hardware (3)AI Research (3)Mathematics (3)Google DeepMind (2)Information Retrieval (2)Open Source AI (2)People (2)AI Ethics (1)AI Infrastructure (1)AI for Science (1)Developer Tools (1)Google (1)Large Language Models (1)MLOps (1)Model Evaluation (1)Programming Languages (1)Statistics (1)Training & Optimization (1)
Showing 1-60 of 116 articles
Attention
Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them.
Attention sink
An attention sink is an empirical phenomenon in Transformer language models in which a large fraction of each attention head's weight concentrates on a few tokens at the very start of the sequence, most often…
Backpropagation
Backpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation.
Batch Normalization
Batch normalization (BatchNorm or BN) is a normalization method used inside some neural networks.
Bayesian Neural Network
A Bayesian neural network (BNN) is a neural network in which the weights and biases are represented as probability distributions rather than fixed point estimates.
Bias
Bias in artificial intelligence carries three distinct technical meanings: a learnable scalar parameter added inside a neuron, the systematic error component of an estimator (the "bias" in the bias-variance…
Bias (Math) or Bias Term
The bias term is a learnable additive constant b added to the weighted sum of a neuron's inputs before an activation function is applied, computed as $$z = w \cdot x + b$$.
Bidirectional
Bidirectional describes a sequence model in which the representation at every position depends on the entire input sequence, both the tokens before it and the tokens after it
Boltzmann machine
A Boltzmann machine is a stochastic, recurrent neural network of symmetrically connected binary units that defines an energy function over its configurations and assigns each configuration a probability…
Calibration Layer
A calibration layer is a post-prediction adjustment appended to a trained machine learning model that rescales its raw output scores or predicted probabilities so they better reflect the true likelihood of…
Candidate Sampling
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose among a very large number of output classes.
Co-Adaptation
Co-adaptation in neural networks refers to a phenomenon in which different hidden units develop highly correlated behavior, becoming excessively dependent on one another rather than learning independent
Connectionism
Connectionism is an approach in cognitive science and artificial intelligence that explains mental phenomena as the collective behavior of large networks of simple, interconnected processing units.
Continual learning
Continual learning, also called lifelong learning or incremental learning, is a machine learning paradigm in which a model learns from a stream of tasks or data distributions over time
ConvNeXt
ConvNeXt is a family of pure convolutional neural network (CNN) models that match or beat Vision Transformers on standard vision benchmarks
Convolutional Neural Network
A convolutional neural network (CNN, or ConvNet) is a neural network that uses convolution-like linear operators in at least some layers.
Convolutional Operation
The convolutional operation is a mathematical procedure that combines two functions to produce a third function expressing how the shape of one is modified by the other.
Cross-encoder
A cross-encoder is a neural network architecture that scores a pair of texts by feeding them jointly into a single transformer (such as BERT) and reading out one scalar score for the pair, for example a…
Decoder
A decoder is the component of a neural network that turns an internal, compressed, or abstract representation into a desired output, such as a translated sentence, a generated image, a reconstructed input, or…
Deep Learning
Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to automatically learn representations of data at multiple levels of abstraction
Deep Neural Network
A deep neural network (DNN) is an artificial neural network with multiple hidden layers of artificial neurons stacked between its input and output layers
DeepNorm / DeepNet
DeepNorm is a normalization and weight initialization scheme for Transformer networks that makes the training of very deep models stable.
DeepSeek Sparse Attention (DSA)
DeepSeek Sparse Attention (DSA) is a trainable, fine-grained sparse attention mechanism introduced by the Chinese AI company DeepSeek in its experimental model DeepSeek-V3.2-Exp, released on September 29, 2025.
Dense Layer
A dense layer, also called a fully connected (FC) layer, linear layer, or affine layer, is a layer in an artificial neural network where every input neuron is connected to every output neuron.
DenseNet
DenseNet (Densely Connected Convolutional Networks) is a convolutional neural network architecture that connects every layer to every other layer in a feed-forward fashion
Depthwise separable convolutional neural network (sepCNN)
A depthwise separable convolutional neural network (often abbreviated sepCNN) is a convolutional neural network that replaces standard convolution layers with depthwise separable convolutions.
Discriminator
A discriminator is the neural network in a generative adversarial network (GAN) that is trained to tell real data apart from data produced by the generator
EfficientNet
EfficientNet is a family of convolutional neural network architectures and a model-scaling method that uniformly scales network depth, width, and input resolution with a single compound coefficient, developed…
Embedding Layer
An embedding layer is a neural network component that acts as a trainable lookup table, mapping discrete integer indices (such as word IDs, user IDs, or category codes) to dense, continuous-valued vectors.
Epoch
An epoch in machine learning is one complete pass through the entire training dataset, during which every example is presented to the model exactly once to compute gradients and update parameters before the…
Expert Choice routing
Expert Choice routing (often abbreviated EC) is a routing method for mixture of experts (MoE) layers in neural networks, introduced by researchers at Google in 2022.
Exploding Gradient Problem
The exploding gradient problem is a training failure in deep neural networks where the gradients of the loss function grow exponentially large as they propagate backward through layers during backpropagation…
Feedforward Neural Network (FFN)
A feedforward neural network (FFN), also called a multilayer perceptron (MLP) when it has multiple layers, is a type of artificial neural network in which information flows in one direction only, from the…
Forget Gate
The forget gate is a sigmoid layer inside a Long Short-Term Memory (LSTM) recurrent neural network that decides, element by element, how much of the previous cell state to keep and how much to discard at each…
Full Softmax
Full softmax (also called the standard softmax or exact softmax) is the softmax computation that calculates a probability for every possible output class in a classification model
Fully Connected Layer
A fully connected layer (also called a dense layer or linear layer) is a layer in an artificial neural network in which every input value connects to every output value through a learned weight
GELU (Gaussian Error Linear Unit)
The Gaussian Error Linear Unit (GELU) is a smooth, non-monotonic activation function defined as GELU(x) = x · Φ(x), where Φ(x) is the cumulative distribution function of the standard normal distribution.
Gated DeltaNet
Gated DeltaNet is a linear attention sequence model that augments the delta rule, a key-value error-correction update used in the earlier DeltaNet architecture
Generative adversarial network
Generative adversarial networks (GANs) are a family of generative models trained through competition between two learned functions.
Generator
A generator is a neural network within a generative adversarial network (GAN) that learns to produce synthetic data samples from random noise.
Graph Neural Network
A Graph Neural Network (GNN) is a class of neural_network that operates directly on graph-structured data, meaning data made of nodes (entities) connected by edges (relationships), by repeatedly passing…
GraphCast
GraphCast is a machine-learning weather-forecasting model from Google DeepMind that produces a 10-day global forecast at 0.25-degree resolution in under one minute on a single Google Cloud TPU v4 device
H-Net (dynamic chunking)
H-Net, short for Hierarchical Network, is a tokenizer-free neural sequence model that learns to segment raw bytes into content-adaptive "chunks" as part of ordinary end-to-end training, rather than relying on…
Hidden Layer
A hidden layer is a layer of artificial neurons in a neural network that sits between the input layer and the output layer
Hopfield Network
A Hopfield network is a form of recurrent neural network that stores information as stable patterns in a system of interconnected binary units and retrieves it by content rather than by address.
Hymba
Hymba is a hybrid-head neural-network architecture for small language models introduced by NVIDIA researchers in November 2024.
Inception (deep learning)
Inception is a family of convolutional neural network (CNN) architectures developed by researchers at Google, first introduced in 2014.
Input Layer
The input layer is the first layer of a neural network: it receives the raw feature vector for each data sample and passes those values forward to the next layer, performing no learned computation of its own.
Intel Loihi
Intel Loihi is a family of research neuromorphic processors developed by Intel Labs to implement spiking neural networks (SNNs) in silicon, with the stated goals of energy-efficient inference, online learning…
John Hopfield
John Joseph Hopfield (born July 15, 1933) is an American physicist whose work spans condensed-matter physics, molecular biology, and the theory of neural computation.
Jürgen Schmidhuber
Jürgen Schmidhuber (anglicized Jurgen Schmidhuber, born 17 January 1963) is a German computer scientist best known as a co-inventor of long short-term memory (LSTM), the recurrent neural network architecture…
Kolmogorov-Arnold Network
A Kolmogorov-Arnold Network (KAN) is a type of neural network architecture proposed as an alternative to the traditional Multi-Layer Perceptron (MLP).
Layer
A layer is the fundamental building block of a neural network: an organized group of neurons (also called nodes or units) that together apply one mathematical transformation to their input and pass the result…
LeNet
LeNet is the pioneering family of convolutional neural networks developed by Yann LeCun and collaborators at AT&T Bell Labs between roughly 1988 and 1998 to read handwritten characters
Lightning Attention
Lightning Attention is an IO-aware (input/output aware) implementation of linear attention that lets the method reach its theoretical linear-time complexity in practice
Linear Probes
A linear probe is a small linear classifier (or linear regressor) trained on the frozen internal activations of a neural network to test whether a particular concept, property, or label is linearly decodable…
Logits
In machine learning and statistics, logits refer to the raw, unnormalized scores output by a model before they are converted into probabilities.
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences.
Mixture of Block Attention (MoBA)
Mixture of Block Attention (MoBA) is a trainable block-sparse attention mechanism introduced in February 2025 by researchers at Moonshot AI, the company that builds the Kimi family of large language models