Neural Networks
116 AI Wiki articles on Neural Networks. The most referenced are Deep Learning, Transformers and Neural Network.
116 articlesRSS
Showing 1-60 of 116 articles
Activation Function
An activation function is a nonlinear mathematical function applied to the output of each neuron in an artificial neural network, and it is what gives the...
Deep LearningMachine Learning
Attention
Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and...
Deep LearningMachine Learning
Attention sink
An attention sink is an empirical phenomenon in Transformer language models in which a large fraction of each attention head's weight concentrates on a few...
Deep Learning
Backpropagation
Backpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation. It...
Deep LearningMachine Learning
Batch Normalization
Batch normalization (BatchNorm or BN) is a normalization method used inside some neural networks. During training, it standardizes each selected activation...
Deep LearningMachine Learning
Bayesian Neural Network
See also: Machine learning terms A Bayesian neural network (BNN) is a neural network in which the weights and biases are represented as probability...
Deep LearningMachine Learning
Bias
See also: Bias-Variance Tradeoff, Fairness (Machine Learning), Neural Network, Activation Function, Weight Bias in artificial intelligence carries three...
AI EthicsMachine Learning
Bias (Math) or Bias Term
See also: Machine learning terms The bias term is a learnable additive constant b added to the weighted sum of a neuron's inputs before an activation function...
Machine LearningMathematics
Bidirectional
See also: Machine learning terms Bidirectional describes a sequence model in which the representation at every position depends on the entire input sequence,...
Model Architecture
Boltzmann machine
A Boltzmann machine is a stochastic, recurrent neural network of symmetrically connected binary units that defines an energy function over its configurations...
AI HistoryGenerative AI
Calibration Layer
A calibration layer is a post-prediction adjustment appended to a trained machine learning model that rescales its raw output scores or predicted probabilities...
Deep LearningMachine Learning
Candidate Sampling
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose...
Machine LearningNatural Language Processing
Co-Adaptation
Co-adaptation in neural networks refers to a phenomenon in which different hidden units develop highly correlated behavior, becoming excessively dependent on...
Deep LearningMachine Learning
Connectionism
Connectionism is an approach in cognitive science and artificial intelligence that explains mental phenomena as the collective behavior of large networks of...
AI HistoryArtificial Intelligence
Continual learning
See also: Transfer learning, Meta-learning, Deep learning, Neural network Continual learning, also called lifelong learning or incremental learning, is a...
Deep LearningMachine Learning
ConvNeXt
ConvNeXt is a family of pure convolutional neural network (CNN) models that match or beat Vision Transformers on standard vision benchmarks, reaching 87.8%...
Computer VisionDeep Learning
Convolutional Neural Network
A convolutional neural network (CNN, or ConvNet) is a neural network that uses convolution-like linear operators in at least some layers. CNNs are especially...
Computer VisionDeep Learning
Convolutional Operation
The convolutional operation is a mathematical procedure that combines two functions to produce a third function expressing how the shape of one is modified by...
Computer VisionMachine Learning
Cross-encoder
A cross-encoder is a neural network architecture that scores a pair of texts by feeding them jointly into a single transformer (such as BERT) and reading out...
Information RetrievalNatural Language Processing
Decoder
See also: Encoder, Transformer, Machine learning terms A decoder is the component of a neural network that turns an internal, compressed, or abstract...
Deep LearningMachine Learning
Deep Learning
Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to automatically learn representations of data at...
Artificial IntelligenceDeep Learning
Deep Neural Network
See also: Machine learning terms A deep neural network (DNN) is an artificial neural network with multiple hidden layers of artificial neurons stacked between...
Deep LearningMachine Learning
DeepNorm / DeepNet
DeepNorm is a normalization and weight initialization scheme for Transformer networks that makes the training of very deep models stable. It was introduced by...
Deep Learning
DeepSeek Sparse Attention (DSA)
DeepSeek Sparse Attention (DSA) is a trainable, fine-grained sparse attention mechanism introduced by the Chinese AI company DeepSeek in its experimental model...
Deep Learning
Dense Layer
A dense layer, also called a fully connected (FC) layer, linear layer, or affine layer, is a layer in an artificial neural network where every input neuron is...
Deep LearningMachine Learning
DenseNet
DenseNet (Densely Connected Convolutional Networks) is a convolutional neural network architecture that connects every layer to every other layer in a...
Computer VisionDeep Learning
Depthwise separable convolutional neural network (sepCNN)
See also: Machine learning terms A depthwise separable convolutional neural network (often abbreviated sepCNN) is a convolutional neural network that replaces...
Computer VisionModel Architecture
Discriminator
A discriminator is the neural network in a generative adversarial network (GAN) that is trained to tell real data apart from data produced by the generator,...
Generative AIMachine Learning
EfficientNet
EfficientNet is a family of convolutional neural network architectures and a model-scaling method that uniformly scales network depth, width, and input...
Computer VisionDeep Learning
Embedding Layer
An embedding layer is a neural network component that acts as a trainable lookup table, mapping discrete integer indices (such as word IDs, user IDs, or...
Machine LearningNatural Language Processing
Epoch
See also: Machine learning terms An epoch in machine learning is one complete pass through the entire training dataset, during which every example is presented...
Deep LearningMachine Learning
Expert Choice routing
Expert Choice routing (often abbreviated EC) is a routing method for mixture of experts (MoE) layers in neural networks, introduced by researchers at Google in...
Deep Learning
Exploding Gradient Problem
See also: Machine learning terms, Vanishing gradient problem, Gradient clipping, Backpropagation The exploding gradient problem is a training failure in deep...
Deep LearningMachine Learning
Feedforward Neural Network (FFN)
A feedforward neural network (FFN), also called a multilayer perceptron (MLP) when it has multiple layers, is a type of artificial neural network in which...
Deep LearningMachine Learning
Forget Gate
The forget gate is a sigmoid layer inside a Long Short-Term Memory (LSTM) recurrent neural network that decides, element by element, how much of the previous...
Machine Learning
Full Softmax
Full softmax (also called the standard softmax or exact softmax) is the softmax computation that calculates a probability for every possible output class in a...
Deep LearningMachine Learning
Fully Connected Layer
A fully connected layer (also called a dense layer or linear layer) is a layer in an artificial neural network in which every input value connects to every...
Deep LearningMachine Learning
GELU (Gaussian Error Linear Unit)
The Gaussian Error Linear Unit (GELU) is a smooth, non-monotonic activation function defined as GELU(x) = x · Φ(x), where Φ(x) is the cumulative distribution...
Artificial IntelligenceDeep Learning
Gated DeltaNet
Gated DeltaNet is a linear attention sequence model that augments the delta rule, a key-value error-correction update used in the earlier DeltaNet...
Deep Learning
Generative adversarial network
Generative adversarial networks (GANs) are a family of generative models trained through competition between two learned functions. A generator transforms...
Deep LearningGenerative AI
Generator
A generator is a neural network within a generative adversarial network (GAN) that learns to produce synthetic data samples from random noise. It maps a point...
Generative AIMachine Learning
Graph Neural Network
A Graph Neural Network (GNN) is a class of neuralnetwork that operates directly on graph-structured data, meaning data made of nodes (entities) connected by...
Deep Learning
GraphCast
GraphCast is a machine-learning weather-forecasting model from Google DeepMind that produces a 10-day global forecast at 0.25-degree resolution in under one...
AI for ScienceGoogle DeepMind
H-Net (dynamic chunking)
H-Net, short for Hierarchical Network, is a tokenizer-free neural sequence model that learns to segment raw bytes into content-adaptive "chunks" as part of...
Deep Learning
Hidden Layer
See also: neural network, input layer, output layer, activation function A hidden layer is a layer of artificial neurons in a neural network that sits between...
Deep LearningMachine Learning
Hopfield Network
A Hopfield network is a form of recurrent neural network that stores information as stable patterns in a system of interconnected binary units and retrieves it...
AI HistoryDeep Learning
Hymba
Hymba is a hybrid-head neural-network architecture for small language models introduced by NVIDIA researchers in November 2024. Its defining idea is to run...
Deep Learning
Inception (deep learning)
Inception is a family of convolutional neural network (CNN) architectures developed by researchers at Google, first introduced in 2014. The original...
Computer VisionDeep Learning
Input Layer
See also: Machine learning terms The input layer is the first layer of a neural network: it receives the raw feature vector for each data sample and passes...
Deep LearningMachine Learning
Intel Loihi
Intel Loihi is a family of research neuromorphic processors developed by Intel Labs to implement spiking neural networks (SNNs) in silicon, with the stated...
AI Hardware
John Hopfield
John Joseph Hopfield (born July 15, 1933) is an American physicist whose work spans condensed-matter physics, molecular biology, and the theory of neural...
AI HistoryDeep Learning
Jürgen Schmidhuber
Jürgen Schmidhuber (anglicized Jurgen Schmidhuber, born 17 January 1963) is a German computer scientist best known as a co-inventor of long short-term memory...
Deep LearningPeople
Kolmogorov-Arnold Network
A Kolmogorov-Arnold Network (KAN) is a type of neural network architecture proposed as an alternative to the traditional Multi-Layer Perceptron (MLP)....
Deep LearningMachine Learning
Layer
A layer is the fundamental building block of a neural network: an organized group of neurons (also called nodes or units) that together apply one mathematical...
Deep LearningMachine Learning
LeNet
LeNet is the pioneering family of convolutional neural networks developed by Yann LeCun and collaborators at AT&T Bell Labs between roughly 1988 and 1998 to...
Artificial IntelligenceComputer Vision
Lightning Attention
Lightning Attention is an IO-aware (input/output aware) implementation of linear attention that lets the method reach its theoretical linear-time complexity in...
Deep Learning
Linear Probes
A linear probe is a small linear classifier (or linear regressor) trained on the frozen internal activations of a neural network to test whether a particular...
Interpretability
Logits
In machine learning and statistics, logits refer to the raw, unnormalized scores output by a model before they are converted into probabilities. The term has...
Deep LearningMachine Learning
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences. An LSTM layer carries two vectors from one time step...
Deep LearningModel Architecture
Mixture of Block Attention (MoBA)
Mixture of Block Attention (MoBA) is a trainable block-sparse attention mechanism introduced in February 2025 by researchers at Moonshot AI, the company that...
Deep Learning