Bahdanau attention
Bahdanau attention is the first attention mechanism for neural networks, introduced in 2014 to let a sequence-to-sequence decoder soft-align to every encoder hidden state instead of relying on a single…
Model Architecture
Explore Model Architecture through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of Model Architecture.
All articles (65)Deep Learning (21)Transformer Models (16)Neural Networks (15)Large Language Models (14)Machine Learning (13)AI Models (6)Microsoft (6)Computer Vision (5)AI Inference (4)Meta AI (4)AI Companies (3)AI Infrastructure (2)Chinese AI (2)Google (2)Mixture of Experts (2)Natural Language Processing (2)Open Source AI (2)AI History (1)Artificial Intelligence (1)Developer Tools (1)Diffusion Models (1)Google DeepMind (1)Image Generation (1)Information Retrieval (1)MLOps (1)Multimodal AI (1)NVIDIA (1)People (1)Training & Optimization (1)
Showing 1-21 of 21 articles
Encoder
An encoder in machine learning is a neural network component that transforms input data (text, an image, audio, or code) into a compressed, structured representation, often called a latent representation…
Hyena
Hyena is a sub-quadratic, attention-free neural sequence operator that replaces the self-attention operator of the Transformer with a recurrence of long, implicitly parameterized convolutions and…
Layer normalization
Layer normalization is a technique for normalizing the activations of a neural network across the feature dimension of each individual sample, rather than across a batch of samples.
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences.
Mamba 2
Mamba 2 is a state space model architecture introduced in the paper "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" by Tri Dao and Albert Gu
Mamba-3
Mamba-3 is a sequence-modeling architecture in the state space model (SSM) family, introduced in March 2026 by researchers at Carnegie Mellon University and Princeton.
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Multilayer Perceptron
A multilayer perceptron (MLP) is a feedforward neural network built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.
Transformers
A Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…
Vision Encoder
A vision encoder is the neural network component that turns an image into a sequence of numerical vectors that other models can consume.
YaRN
YaRN (Yet another RoPE extensioN) is a compute-efficient method for extending the context window of large language models that use Rotary Position Embeddings (RoPE).
xLSTM
xLSTM (Extended Long Short-Term Memory) is a recurrent neural network architecture introduced in May 2024 by Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, and collaborators at Johannes Kepler University…