Showing 1-60 of 112 articles
Activation Function
An activation function is a nonlinear mathematical function applied to the output of each neuron in an artificial neural network, and it is what gives the...
Deep LearningMachine Learning
Attention
See also: Machine learning terms > This article gives a high-level overview of attention as a family of mechanisms in machine learning. For deeper treatments,...
Deep LearningMachine Learning
Attention sink
An attention sink is an empirical phenomenon in Transformer language models in which a large fraction of each attention head's weight concentrates on a few...
Deep Learning
Backpropagation
Backpropagation (short for "backward propagation of errors") is the algorithm used to compute the gradient of a scalar loss function with respect to every...
Deep LearningMachine Learning
Batch Normalization
See also: Machine learning terms Batch normalization (often abbreviated BatchNorm or BN) is a technique for improving the speed, stability, and performance of...
Deep LearningMachine Learning
Bayesian Neural Network
A Bayesian neural network (BNN) is a neural network in which the weights and biases are represented as probability distributions rather than fixed point...
Deep LearningMachine Learning
Bias
Bias in artificial intelligence carries three distinct technical meanings: a learnable scalar parameter added inside a neuron, the systematic error component...
AI EthicsMachine Learning
Bias (Math) or Bias Term
See also: Machine learning terms The bias term is a learnable additive constant b added to the weighted sum of a neuron's inputs before an activation function...
Machine LearningMathematics
Bidirectional
See also: Machine learning terms Bidirectional describes a sequence model in which the representation at every position depends on the entire input sequence,...
Model Architecture
Boltzmann machine
A Boltzmann machine is a stochastic, recurrent neural network of symmetrically connected binary units that defines an energy function over its configurations...
AI HistoryGenerative AI
Calibration Layer
A calibration layer is a post-prediction adjustment appended to a trained machine learning model that rescales its raw output scores or predicted probabilities...
Deep LearningMachine Learning
Candidate Sampling
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose...
Machine LearningNatural Language Processing
Co-Adaptation
Co-adaptation in neural networks refers to a phenomenon in which different hidden units develop highly correlated behavior, becoming excessively dependent on...
Deep LearningMachine Learning
Continual learning
Continual learning, also called lifelong learning or incremental learning, is a machine learning paradigm in which a model learns from a stream of tasks or...
Deep LearningMachine Learning
ConvNeXt
ConvNeXt is a family of pure convolutional neural network (CNN) models that match or beat Vision Transformers on standard vision benchmarks, reaching 87.8%...
Computer VisionDeep Learning
Convolutional Neural Network
A convolutional neural network (CNN or ConvNet) is a type of neural network that processes grid-like data such as images by sliding small learnable filters...
Computer VisionDeep Learning
Convolutional Operation
The convolutional operation is a mathematical procedure that combines two functions to produce a third function expressing how the shape of one is modified by...
Computer VisionMachine Learning
Cross-encoder
A cross-encoder is a neural network architecture that scores a pair of texts by feeding them jointly into a single transformer (such as BERT) and reading out...
Information RetrievalNatural Language Processing
Decoder
See also: Encoder, Transformer, Machine learning terms A decoder is the component of a neural network that turns an internal, compressed, or abstract...
Deep LearningMachine Learning
Deep Learning
Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to automatically learn representations of data at...
Artificial IntelligenceDeep Learning
Deep Neural Network
A deep neural network (DNN) is an artificial neural network with multiple hidden layers of artificial neurons stacked between its input and output layers,...
Deep LearningMachine Learning
DeepNorm / DeepNet
DeepNorm is a normalization and weight initialization scheme for Transformer networks that makes the training of very deep models stable. It was introduced by...
Deep Learning
DeepSeek Sparse Attention (DSA)
DeepSeek Sparse Attention (DSA) is a trainable, fine-grained sparse attention mechanism introduced by the Chinese AI company DeepSeek in its experimental model...
Deep Learning
Dense Layer
A dense layer, also called a fully connected (FC) layer, linear layer, or affine layer, is a layer in an artificial neural network where every input neuron is...
Deep LearningMachine Learning
DenseNet
DenseNet (Densely Connected Convolutional Networks) is a convolutional neural network architecture that connects every layer to every other layer in a...
Computer VisionDeep Learning
Depthwise separable convolutional neural network (sepCNN)
See also: Machine learning terms A depthwise separable convolutional neural network (often abbreviated sepCNN) is a convolutional neural network that replaces...
Computer VisionModel Architecture
Discriminator
A discriminator is the neural network in a generative adversarial network (GAN) that is trained to tell real data apart from data produced by the generator,...
Generative AIMachine Learning
EfficientNet
EfficientNet is a family of convolutional neural network architectures and a model-scaling method that uniformly scales network depth, width, and input...
Computer VisionDeep Learning
Embedding Layer
An embedding layer is a neural network component that acts as a trainable lookup table, mapping discrete integer indices (such as word IDs, user IDs, or...
Machine LearningNatural Language Processing
Epoch
An epoch in machine learning is one complete pass through the entire training dataset, during which every example is presented to the model exactly once to...
Deep LearningMachine Learning
Expert Choice routing
Expert Choice routing (often abbreviated EC) is a routing method for mixture of experts (MoE) layers in neural networks, introduced by researchers at Google in...
Deep Learning
Exploding Gradient Problem
See also: Machine learning terms, Vanishing gradient problem, Gradient clipping, Backpropagation The exploding gradient problem is a training failure in deep...
Deep LearningMachine Learning
Feedforward Neural Network (FFN)
A feedforward neural network (FFN), also called a multilayer perceptron (MLP) when it has multiple layers, is a type of artificial neural network in which...
Deep LearningMachine Learning
Forget Gate
The forget gate is a sigmoid layer inside a Long Short-Term Memory (LSTM) recurrent neural network that decides, element by element, how much of the previous...
Machine Learning
Full Softmax
Full softmax (also called the standard softmax or exact softmax) is the softmax computation that calculates a probability for every possible output class in a...
Deep LearningMachine Learning
Fully Connected Layer
A fully connected layer (also called a dense layer or linear layer) is a layer in an artificial neural network in which every input value connects to every...
Deep LearningMachine Learning
GELU (Gaussian Error Linear Unit)
The Gaussian Error Linear Unit (GELU) is a smooth, non-monotonic activation function defined as GELU(x) = x · Φ(x), where Φ(x) is the cumulative distribution...
Artificial IntelligenceDeep Learning
Gated DeltaNet
Gated DeltaNet is a linear attention sequence model that augments the delta rule, a key-value error-correction update used in the earlier DeltaNet...
Deep Learning
Generative adversarial network
A generative adversarial network (GAN) is a class of machine learning model in which two neural networks, a generator and a discriminator, are trained...
Deep LearningGenerative AI
Generator
A generator is a neural network within a generative adversarial network (GAN) that learns to produce synthetic data samples from random noise. It maps a point...
Generative AIMachine Learning
Graph Neural Network
A Graph Neural Network (GNN) is a class of neuralnetwork that operates directly on graph-structured data, meaning data made of nodes (entities) connected by...
Deep Learning
GraphCast
GraphCast is a machine-learning weather-forecasting model from Google DeepMind that produces a 10-day global forecast at 0.25-degree resolution in under one...
AI for ScienceGoogle DeepMind
H-Net (dynamic chunking)
H-Net, short for Hierarchical Network, is a tokenizer-free neural sequence model that learns to segment raw bytes into content-adaptive "chunks" as part of...
Deep Learning
Hidden Layer
See also: neural network, input layer, output layer, activation function A hidden layer is a layer of artificial neurons in a neural network that sits between...
Deep LearningMachine Learning
Hymba
Hymba is a hybrid-head neural-network architecture for small language models introduced by NVIDIA researchers in November 2024. Its defining idea is to run...
Deep Learning
Inception (deep learning)
Inception is a family of convolutional neural network (CNN) architectures developed by researchers at Google, first introduced in 2014. The original...
Computer VisionDeep Learning
Input Layer
The input layer is the first layer of a neural network: it receives the raw feature vector for each data sample and passes those values forward to the next...
Deep LearningMachine Learning
Intel Loihi
Intel Loihi is a family of research neuromorphic processors developed by Intel Labs to implement spiking neural networks (SNNs) in silicon, with the stated...
AI Hardware
Jürgen Schmidhuber
Jürgen Schmidhuber (anglicized Jurgen Schmidhuber, born 17 January 1963) is a German computer scientist best known as a co-inventor of long short-term memory...
Deep LearningPeople
Kolmogorov-Arnold Network
A Kolmogorov-Arnold Network (KAN) is a type of neural network architecture proposed as an alternative to the traditional Multi-Layer Perceptron (MLP)....
Deep LearningMachine Learning
LSTM
Long Short-Term Memory (LSTM) is a recurrent neural network architecture that learns long-range dependencies in sequential data by maintaining a separate,...
Deep LearningModel Architecture
Layer
A layer is the fundamental building block of a neural network: an organized group of neurons (also called nodes or units) that together apply one mathematical...
Deep LearningMachine Learning
LeNet
LeNet is the pioneering family of convolutional neural networks developed by Yann LeCun and collaborators at AT&T Bell Labs between roughly 1988 and 1998 to...
Artificial IntelligenceComputer Vision
Lightning Attention
Lightning Attention is an IO-aware (input/output aware) implementation of linear attention that lets the method reach its theoretical linear-time complexity in...
Deep Learning
Linear Probes
A linear probe is a small linear classifier (or linear regressor) trained on the frozen internal activations of a neural network to test whether a particular...
Interpretability
Logits
In machine learning and statistics, logits refer to the raw, unnormalized scores output by a model before they are converted into probabilities. The term has...
Deep LearningMachine Learning
Long Short-Term Memory (LSTM)
Long Short-Term Memory (LSTM) is a specialized type of recurrent neural network (RNN) architecture designed to learn long-range dependencies in sequential...
Deep LearningMachine Learning
Mixture of Block Attention (MoBA)
Mixture of Block Attention (MoBA) is a trainable block-sparse attention mechanism introduced in February 2025 by researchers at Moonshot AI, the company that...
Deep Learning
Mixture of Experts (MoE)
A Mixture of Experts (MoE) is a machine learning architecture that divides a problem into subtasks, each handled by a specialized sub-network called an...
Deep LearningMachine Learning
Mixture-of-Recursions (MoR)
Mixture-of-Recursions (MoR) is a Transformer architecture, introduced in 2025, that unifies two previously separate strategies for building efficient language...
Deep Learning