ALiBi (Attention with Linear Biases)
ALiBi (Attention with Linear Biases) is a positional encoding method for transformer language models that, instead of adding positional embeddings to word vectors, adds a static, non-learned penalty to…
Model Architecture
Explore Model Architecture through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Transformer Models
Articles that also belong to these categories. Counts cover all of Model Architecture.
All articles (65)Deep Learning (21)Transformer Models (16)Neural Networks (15)Large Language Models (14)Machine Learning (13)AI Models (6)Microsoft (6)Computer Vision (5)AI Inference (4)Meta AI (4)AI Companies (3)AI Infrastructure (2)Chinese AI (2)Google (2)Mixture of Experts (2)Natural Language Processing (2)Open Source AI (2)AI History (1)Artificial Intelligence (1)Developer Tools (1)Diffusion Models (1)Google DeepMind (1)Image Generation (1)Information Retrieval (1)MLOps (1)Multimodal AI (1)NVIDIA (1)People (1)Training & Optimization (1)
Showing 1-16 of 16 articles
Cross-attention
Cross-attention is a variant of the attention mechanism in which the queries are derived from one sequence or representation while the keys and values are derived from a different sequence or representation
Differential Transformer
The Differential Transformer (often shortened to Diff Transformer or DIFF Transformer) is a decoder-only neural sequence architecture introduced by researchers at Microsoft Research and Tsinghua University in…
Infini-Attention
Infini-attention is an attention mechanism introduced by Google researchers Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal in the April 2024 paper "Leave No Context Behind: Efficient Infinite…
Linear Attention
Linear attention is a family of sub-quadratic attention mechanisms that replaces the softmax dot-product operation of standard Transformer self-attention with a feature-map-based factorization
LongNet
LongNet is a transformer variant introduced by Microsoft Research in July 2023 that is designed to scale attention to sequences exceeding one billion tokens while preserving performance on shorter inputs.
MEGABYTE
MEGABYTE is a transformer architecture for autoregressive modeling of very long sequences directly at the byte level, introduced by researchers at Meta AI (FAIR) in May 2023.
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-Query Attention (MQA)
Multi-Query Attention (MQA) is a variant of the multi-head attention mechanism used in transformer neural networks in which all query heads share a single key head and a single value head, rather than each…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
RMSNorm
RMSNorm (Root Mean Square Layer Normalization) is a feature normalization technique introduced by Biao Zhang and Rico Sennrich in 2019 that scales each activation vector by its root mean square only, dropping…
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sliding window attention
Sliding window attention (SWA) is a sparse attention pattern in which each query token attends only to a fixed-size window of nearby tokens instead of to every preceding (or every other) token in the sequence.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence