Mixture of Experts (MoE)
A mixture of experts (MoE) is a machine-learning architecture that contains multiple component functions called experts and a gating or routing mechanism that determines how their outputs contribute to a…
Neural Networks
Explore Neural Networks through related topics and the articles other pages reference most.
Explore articles
Browse subtopics (28)
Articles that also belong to these categories. Counts cover all of Neural Networks.
All articles (116)Deep Learning (81)Machine Learning (58)Computer Vision (17)Model Architecture (15)Interpretability (6)AI History (5)Artificial Intelligence (4)Generative AI (4)Natural Language Processing (4)Transformer Models (4)AI Hardware (3)AI Research (3)Mathematics (3)Google DeepMind (2)Information Retrieval (2)Open Source AI (2)People (2)AI Ethics (1)AI Infrastructure (1)AI for Science (1)Developer Tools (1)Google (1)Large Language Models (1)MLOps (1)Model Evaluation (1)Programming Languages (1)Statistics (1)Training & Optimization (1)
Showing 61-116 of 116 articles
Mixture-of-Recursions (MoR)
Mixture-of-Recursions (MoR) is a Transformer architecture, introduced in 2025, that unifies two previously separate strategies for building efficient language models: parameter sharing through recursion, and…
MobileNet
MobileNet is a family of efficient convolutional neural network (CNN) architectures developed by Google for mobile and edge AI applications.
Monosemanticity
Monosemanticity is a property of an internal feature or neuron in a neural network when that unit responds to a single, human-interpretable concept rather than to a heterogeneous collection of unrelated inputs.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Multilayer Perceptron
A multilayer perceptron (MLP) is a feedforward neural network built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer
Native Sparse Attention (NSA)
Native Sparse Attention (NSA) is a hardware-aligned, natively trainable sparse attention mechanism introduced in February 2025 by DeepSeek, in collaboration with researchers at Peking University and the…
NeRF
Neural Radiance Fields (NeRF) is a method for synthesizing photorealistic novel views of a 3D scene by encoding the scene as a continuous 5D function (3D position plus 2D viewing direction) inside a single…
Neural Network
A neural network (also called an artificial neural network or ANN) is a computational model, loosely inspired by the networks of biological neurons in animal brains
Neural Tangent Kernel
The neural tangent kernel (NTK) is a kernel function built from the parameter gradients of a neural network.
Neural architecture search
Neural architecture search (NAS) is a technique for automating the design of neural network architectures.
Neuromorphic computing
Neuromorphic computing is brain-inspired computer hardware that processes information with spiking neural networks (SNNs) and event-driven, in-memory computation, co-locating memory and processing the way…
Neuron
A neuron (also called a node or unit) is the fundamental computational element of an artificial neural network: it takes one or more numeric inputs, multiplies each by a learned weight, sums them with a bias…
Node (neural network)
A node in a neural network is the basic computational element, an artificial neuron, that receives one or more inputs, multiplies each by a learned weight, sums them, adds a bias, and passes the result through…
Output Layer
The output layer is the final layer of a neural network: it takes the features computed by the hidden layers and converts them into the model's prediction, with its size and activation function fixed by the…
PEER (Parameter Efficient Expert Retrieval / Mixture of a Million Experts)
PEER, short for Parameter Efficient Expert Retrieval, is a neural network layer for Transformer models that replaces the dense feed-forward block with a sparse mixture of experts drawn from an extremely large…
Parameter
In machine learning and statistics, a parameter is an internal variable of a model whose value is learned from data during the training process.
Perceiver
Perceiver is a family of general-purpose neural network architectures from DeepMind built around attention and a small latent bottleneck.
Perceptron
A perceptron is the earliest trainable artificial neural network: a single-layer linear model that classifies inputs into two categories by computing a weighted sum and applying a step threshold.
Polysemanticity
Polysemanticity is the phenomenon in artificial neural networks in which a single neuron (or directional unit such as an attention head) activates strongly for multiple, semantically unrelated inputs or…
Pooling
Pooling is a downsampling operation in neural networks that aggregates each local region of a feature map into a single summary value
Position Interpolation (PI)
Position Interpolation (PI) is a method for extending the context window of a pretrained large language model that uses rotary position embedding (RoPE).
RNN
RNN is the standard abbreviation for recurrent neural network, a class of artificial neural network in which connections between units form cycles
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
RWKV-7 (Goose)
RWKV-7, codenamed Goose, is an attention-free, RNN-style large-language-model architecture introduced in March 2025 that runs inference in linear time with constant memory per token while still training in…
ReLU
ReLU, short for rectified linear unit, is an activation function that maps a real input to its positive part:
Rectified Linear Unit (ReLU)
The Rectified Linear Unit (ReLU) is the most widely used activation function in deep learning, defined mathematically as $$f(x) = \max(0, x)$$: it returns the input directly when positive and outputs zero…
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Register tokens (Vision Transformers Need Registers)
Register tokens are a small set of extra learnable tokens added to the input sequence of a Vision Transformer (ViT) so the network has a dedicated place to carry out internal, image-level computation.
ResNet
ResNet, short for residual network, is a family of deep convolutional neural networks introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at CVPR 2016.
Retentive Network (RetNet)
RetNet (Retentive Network) is a sequence-modeling architecture proposed by Microsoft Research and Tsinghua University in July 2023 as a successor to the Transformer for large language models.
Self-Extend
Self-Extend (written SelfExtend in the original paper) is a training-free technique that lets a pretrained large language model process inputs much longer than the context window it was trained on, with no…
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sigmoid Function
The sigmoid function is a mathematical function that maps any real number to a value between 0 and 1 using the formula $$\sigma(x) = \frac{1}{1 + e^{-x}}$$, producing a smooth S-shaped curve.
Soft MoE
Soft MoE (Soft Mixture of Experts) is a fully differentiable variant of the sparse mixture of experts (MoE) layer.
Softmax
Softmax is a function that converts a finite vector of real-valued scores into a vector of positive numbers that sum to one.
Sparse Coding
Sparse coding is a representation learning principle in which a signal is encoded as a linear combination of a small number of elements drawn from a (usually overcomplete) dictionary of atoms.
Spatial Pooling
Spatial pooling is a downsampling operation in convolutional neural networks (CNNs) that replaces a local region of a feature map with a single summary statistic, such as the maximum or the average of the…
Superposition (Mechanistic Interpretability)
Superposition is the phenomenon in which an artificial neural network represents more distinct features than it has dimensions in its activation space, by assigning those features to nearly-orthogonal (rather…
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping windows and introduces a shifted window…
Tanh (hyperbolic tangent)
The hyperbolic tangent, written tanh, is a smooth, S-shaped activation function that maps any real number into the open interval $$(-1, 1)$$, passing through the origin so that tanh(0) = 0 .
Titans (neural architecture)
Titans is a family of neural sequence-modeling architectures from Google Research that combines an attention-based "short-term memory" with a deep neural "long-term memory" module that updates its own…
Transcoder
A transcoder is a sparse neural network used in mechanistic interpretability research to approximate the input-to-output function of a component inside a transformer (most commonly an MLP sublayer) using a…
Transformers
A Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…
Translational invariance
Translational invariance (also called translation invariance or shift invariance) is the property of a function, system, or machine learning model whose output does not change when its input is translated…
Two-Tower Model
The two-tower model, also known as the dual encoder, bi-encoder, or Siamese network for retrieval, is a neural network architecture that encodes a query and a candidate item with two separate sub-networks…
VGG
VGG (also called VGGNet) is a deep convolutional neural network architecture, introduced in 2014 by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group at the University of Oxford
Vanishing Gradient Problem
The vanishing gradient problem is a difficulty in training deep neural networks where the gradients used to update the network shrink exponentially as they are propagated backward through the layers, leaving…
Weight
In machine learning and neural networks, a weight is a learnable numerical parameter that determines the strength of the connection between two neurons.
Weighted Sum
A weighted sum is a mathematical operation that combines multiple input values by multiplying each value by a corresponding weight (coefficient) and then summing the results.
Wide Model
A wide model is a type of machine learning model that uses a large number of input features, often with sparse, high-dimensional representations such as one-hot encoding and cross-product feature…
YOLO (object detection)
YOLO (You Only Look Once) is a family of object detection models that treat detection as a single regression problem, predicting bounding boxes and class probabilities directly from full images in one forward…
tf.keras
tf.keras is the high-level deep learning API built directly into the TensorFlow machine learning framework
xLSTM
xLSTM (Extended Long Short-Term Memory) is a recurrent neural network architecture introduced in May 2024 by Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, and collaborators at Johannes Kepler University…