Bahdanau attention
Bahdanau attention is the first attention mechanism for neural networks, introduced in 2014 to let a sequence-to-sequence decoder soft-align to every encoder hidden state instead of relying on a single…
Deep Learning
Explore Deep Learning through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Model Architecture
Articles that also belong to these categories. Counts cover all of Deep Learning.
All articles (334)Machine Learning (188)Neural Networks (81)Computer Vision (53)Training & Optimization (41)Natural Language Processing (37)Generative AI (31)Model Architecture (22)Transformer Models (17)Artificial Intelligence (16)Large Language Models (12)People (12)Reinforcement Learning (12)AI Inference (11)Developer Tools (11)Mathematics (11)Interpretability (8)AI Infrastructure (7)Speech & Audio AI (7)AI Hardware (6)AI History (5)AI Research (5)Google DeepMind (5)Open Source AI (5)AI Companies (4)AI Compute (4)AI for Science (4)Algorithms (4)Model Evaluation (4)Robotics (4)AI Models (3)Google (3)Meta AI (3)OpenAI (3)AI Alignment (2)AI Tools & Products (2)Chinese AI (2)Data & Datasets (2)Healthcare AI (2)MLOps (2)Microsoft (2)Multimodal AI (2)NVIDIA (2)Software Development (2)Statistics (2)AI Benchmarks (1)AI Ethics (1)AI Safety (1)Data Science (1)Diffusion Models (1)Embodied AI (1)Image Generation (1)Information Retrieval (1)Programming Languages (1)Prompt Engineering (1)Research Organizations (1)Video Generation (1)
Showing 1-22 of 22 articles
Encoder
An encoder in machine learning is a neural network component that transforms input data (text, an image, audio, or code) into a compressed, structured representation, often called a latent representation…
Hyena
Hyena is a sub-quadratic, attention-free neural sequence operator that replaces the self-attention operator of the Transformer with a recurrence of long, implicitly parameterized convolutions and…
Layer normalization
Layer normalization is a technique for normalizing the activations of a neural network across the feature dimension of each individual sample, rather than across a batch of samples.
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences.
Mamba 2
Mamba 2 is a state space model architecture introduced in the paper "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" by Tri Dao and Albert Gu
Mamba-3
Mamba-3 is a sequence-modeling architecture in the state space model (SSM) family, introduced in March 2026 by researchers at Carnegie Mellon University and Princeton.
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that compresses the keys and values of all attention heads into one shared low-rank latent vector, so that only this vector
Multilayer Perceptron
A multilayer perceptron (MLP) is a feedforward neural network built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer
Native Sparse Attention (NSA)
Native Sparse Attention (NSA) is a hardware-aligned, natively trainable sparse attention mechanism introduced in February 2025 by researchers at DeepSeek, Peking University and the University of Washington.
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.
Test-Time Training (TTT)
Test-Time Training (TTT) is a family of machine learning techniques in which a model updates some of its own parameters at inference time
Transformers
A Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…
Vision Encoder
A vision encoder is the neural network component that turns an image into a sequence of numerical vectors that other models can consume.
YaRN
YaRN (Yet another RoPE extensioN method) is a compute-efficient method for extending the context window of large language models that use Rotary Position Embeddings (RoPE).
xLSTM
xLSTM (Extended Long Short-Term Memory) is a recurrent neural network architecture introduced in May 2024 by Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, and collaborators at Johannes Kepler University…