Showing 61-112 of 112 articles
MobileNet
MobileNet is a family of efficient convolutional neural network (CNN) architectures developed by Google for mobile and edge AI applications. First introduced...
AI HardwareComputer Vision
Monosemanticity
Monosemanticity is a property of an internal feature or neuron in a neural network when that unit responds to a single, human-interpretable concept rather than...
Interpretability
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations...
Deep LearningMachine Learning
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or...
Deep LearningMachine Learning
Native Sparse Attention (NSA)
Native Sparse Attention (NSA) is a hardware-aligned, natively trainable sparse attention mechanism introduced in February 2025 by DeepSeek, in collaboration...
Deep Learning
NeRF
Neural Radiance Fields (NeRF) is a method for synthesizing photorealistic novel views of a 3D scene by encoding the scene as a continuous 5D function (3D...
Computer VisionDeep Learning
Neural Network
A neural network (also called an artificial neural network or ANN) is a computational model, loosely inspired by the networks of biological neurons in animal...
Deep LearningMachine Learning
Neural architecture search
Neural architecture search (NAS) is a technique for automating the design of neural network architectures. Rather than relying on human experts to hand-craft...
Deep LearningMLOps
Neuromorphic computing
Neuromorphic computing is brain-inspired computer hardware that processes information with spiking neural networks (SNNs) and event-driven, in-memory...
AI HardwareAI Infrastructure
Neuron
A neuron (also called a node or unit) is the fundamental computational element of an artificial neural network: it takes one or more numeric inputs, multiplies...
Machine Learning
Node (neural network)
See also: Machine learning terms A node in a neural network is the basic computational element, an artificial neuron, that receives one or more inputs,...
Model Architecture
Output Layer
See also: neural network, activation function, loss function, hidden layer, softmax, backpropagation The output layer is the final layer of a neural network:...
Deep LearningMachine Learning
PEER (Parameter Efficient Expert Retrieval / Mixture of a Million Experts)
PEER, short for Parameter Efficient Expert Retrieval, is a neural network layer for Transformer models that replaces the dense feed-forward block with a sparse...
Deep Learning
Parameter
In machine learning and statistics, a parameter is an internal variable of a model whose value is learned from data during the training process.[1] Parameters...
Machine Learning
Perceiver
Perceiver is a family of general-purpose neural network architectures from DeepMind built around attention and a small latent bottleneck. The first model,...
Google DeepMindModel Architecture
Perceptron
A perceptron is the earliest trainable artificial neural network: a single-layer linear model that classifies inputs into two categories by computing a...
Machine Learning
Polysemanticity
Polysemanticity is the phenomenon in artificial neural networks in which a single neuron (or directional unit such as an attention head) activates strongly for...
Interpretability
Pooling
See also: convolutional neural network, convolutional layer, feature map, downsampling Pooling is a downsampling operation in neural networks that aggregates...
Computer Vision
Position Interpolation (PI)
Position Interpolation (PI) is a method for extending the context window of a pretrained large language model that uses rotary position embedding (RoPE)....
Deep Learning
RNN
See also: Machine learning terms RNN is the standard abbreviation for recurrent neural network, a class of artificial neural network in which connections...
Deep Learning
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time,...
Deep LearningMachine Learning
RWKV-7 (Goose)
RWKV-7, codenamed Goose, is an attention-free, RNN-style large-language-model architecture introduced in March 2025 that runs inference in linear time with...
AI ResearchLarge Language Models
ReLU
See also: Machine learning terms ReLU (Rectified Linear Unit) is an activation function used in neural networks, defined by the formula : it passes positive...
Deep LearningMachine Learning
Rectified Linear Unit (ReLU)
See also: Activation function, Deep learning, Neural network The Rectified Linear Unit (ReLU) is the most widely used activation function in deep learning,...
Deep LearningMachine Learning
Recurrent Neural Network
See also: Machine learning terms A recurrent neural network (RNN) is a class of artificial neural network designed to process sequential data by maintaining an...
Deep LearningMachine Learning
Register tokens (Vision Transformers Need Registers)
Register tokens are a small set of extra learnable tokens added to the input sequence of a Vision Transformer (ViT) so the network has a dedicated place to...
Deep Learning
ResNet
ResNet (Residual Network) is a deep convolutional neural network architecture introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, then at...
Computer VisionDeep Learning
Retentive Network (RetNet)
See also: Transformer, Mamba, RWKV, Linear Attention, Microsoft Research RetNet (Retentive Network) is a sequence-modeling architecture proposed by Microsoft...
AI ResearchOpen Source AI
Self-Extend
Self-Extend (written SelfExtend in the original paper) is a training-free technique that lets a pretrained large language model process inputs much longer than...
Deep Learning
Self-attention
Self-attention is a mechanism that lets a neural network weigh how much every element of a single input sequence should influence every other element,...
Deep LearningMachine Learning
Sigmoid Function
See also: Machine learning terms The sigmoid function is a mathematical function that maps any real number to a value between 0 and 1 using the formula ,...
Deep LearningMachine Learning
Soft MoE
Soft MoE (Soft Mixture of Experts) is a fully differentiable variant of the sparse mixture of experts (MoE) layer. Instead of routing each token to a small set...
Deep Learning
Softmax
See also: Machine learning terms The softmax function is a mathematical function that converts a vector of real numbers, often called logits, into a...
Deep LearningMachine Learning
Sparse Coding
Sparse coding is a representation learning principle in which a signal is encoded as a linear combination of a small number of elements drawn from a (usually...
InterpretabilityMachine Learning
Spatial Pooling
Spatial pooling is a downsampling operation in convolutional neural networks (CNNs) that replaces a local region of a feature map with a single summary...
Computer VisionMachine Learning
Superposition (Mechanistic Interpretability)
Superposition is the phenomenon in which an artificial neural network represents more distinct features than it has dimensions in its activation space, by...
Interpretability
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including...
Deep LearningModel Architecture
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping...
Computer VisionDeep Learning
Tanh (hyperbolic tangent)
The hyperbolic tangent, written tanh, is a smooth, S-shaped activation function that maps any real number into the open interval , passing through the origin...
Deep Learning
Titans (neural architecture)
Titans is a family of neural sequence-modeling architectures from Google Research that combines an attention-based "short-term memory" with a deep neural...
GoogleModel Architecture
Transcoder
A transcoder is a sparse neural network used in mechanistic interpretability research to approximate the input-to-output function of a component inside a...
Interpretability
Transformers
> Note: This article is about the neural network architecture introduced in 2017. For the open-source Python library by Hugging Face, see Hugging Face...
Deep LearningModel Architecture
Translational invariance
Translational invariance (also called translation invariance or shift invariance) is the property of a function, system, or machine learning model whose output...
Computer VisionDeep Learning
Two-Tower Model
The two-tower model, also known as the dual encoder, bi-encoder, or Siamese network for retrieval, is a neural network architecture that encodes a query and a...
Information RetrievalMachine Learning
VGG
VGG (also called VGGNet) is a deep convolutional neural network architecture, introduced in 2014 by Karen Simonyan and Andrew Zisserman of the Visual Geometry...
Computer VisionDeep Learning
Vanishing Gradient Problem
See also: Machine learning terms The vanishing gradient problem is a difficulty in training deep neural networks where the gradients used to update the network...
Deep LearningMachine Learning
Weight
In machine learning and neural networks, a weight is a learnable numerical parameter that determines the strength of the connection between two neurons....
Machine Learning
Weighted Sum
A weighted sum is a mathematical operation that combines multiple input values by multiplying each value by a corresponding weight (coefficient) and then...
Machine LearningMathematics
Wide Model
See also: Machine learning, Deep learning, Recommendation system A wide model is a type of machine learning model that uses a large number of input features,...
Machine Learning
YOLO (object detection)
YOLO (You Only Look Once) is a family of object detection models that treat detection as a single regression problem, predicting bounding boxes and class...
Computer VisionDeep Learning
tf.keras
tf.keras is the high-level deep learning API built directly into the TensorFlow machine learning framework, and it has been TensorFlow's official and...
Deep LearningDeveloper Tools
xLSTM
xLSTM (Extended Long Short-Term Memory) is a recurrent neural network architecture introduced in May 2024 by Maximilian Beck, Korbinian Pöppel, Sepp...
Deep LearningModel Architecture