ALiBi (Attention with Linear Biases)
ALiBi (Attention with Linear Biases) is a positional encoding method for transformer language models that, instead of adding positional embeddings to word vectors, adds a static, non-learned penalty to…
Model Architecture
Explore Model Architecture through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (29)
Articles that also belong to these categories. Counts cover all of Model Architecture.
All articles (65)Deep Learning (21)Transformer Models (16)Neural Networks (15)Large Language Models (14)Machine Learning (13)AI Models (6)Microsoft (6)Computer Vision (5)AI Inference (4)Meta AI (4)AI Companies (3)AI Infrastructure (2)Chinese AI (2)Google (2)Mixture of Experts (2)Natural Language Processing (2)Open Source AI (2)AI History (1)Artificial Intelligence (1)Developer Tools (1)Diffusion Models (1)Google DeepMind (1)Image Generation (1)Information Retrieval (1)MLOps (1)Multimodal AI (1)NVIDIA (1)People (1)Training & Optimization (1)
Showing 1-60 of 65 articles
Albert Gu
Albert Gu is an American computer scientist, Assistant Professor of Machine Learning at Carnegie Mellon University, and co-founder and Chief Scientist of Cartesia AI.
AutoML (Automated Machine Learning)
AutoML (Automated Machine Learning) is the automation of the end-to-end pipeline of applying machine learning to real-world data, replacing manual trial and error with a formal search over data preprocessing…
Bahdanau attention
Bahdanau attention is the first attention mechanism for neural networks, introduced in 2014 to let a sequence-to-sequence decoder soft-align to every encoder hidden state instead of relying on a single…
Bidirectional
Bidirectional describes a sequence model in which the representation at every position depends on the entire input sequence, both the tokens before it and the tokens after it
BitNet
BitNet is a family of large language model architectures developed by Microsoft Research Asia that constrain the weights of a transformer to extremely low bit-widths: initially a single bit ({-1, +1}) and…
BitNet b1.58
BitNet b1.58 is a ternary-weight large language model architecture from Microsoft Research in which every weight is constrained to one of three values, -1, 0, or +1
Byte Latent Transformer
The Byte Latent Transformer (BLT) is a tokenizer-free large language model architecture introduced by researchers at Meta AI's Fundamental AI Research (FAIR) group in December 2024.
Cross-attention
Cross-attention is a variant of the attention mechanism in which the queries are derived from one sequence or representation while the keys and values are derived from a different sequence or representation
Depthwise Separable CNN
A depthwise separable convolution is a factorized form of convolution that decomposes a standard convolutional operation into two sequential steps: a depthwise convolution and a pointwise convolution.
Depthwise separable convolutional neural network (sepCNN)
A depthwise separable convolutional neural network (often abbreviated sepCNN) is a convolutional neural network that replaces standard convolution layers with depthwise separable convolutions.
Differential Transformer
The Differential Transformer (often shortened to Diff Transformer or DIFF Transformer) is a decoder-only neural sequence architecture introduced by researchers at Microsoft Research and Tsinghua University in…
Encoder
An encoder in machine learning is a neural network component that transforms input data (text, an image, audio, or code) into a compressed, structured representation, often called a latent representation…
Feature Pyramid Network (FPN)
A Feature Pyramid Network (FPN) is a generic feature-extraction architecture for object detection and other dense-prediction tasks that builds a multi-scale feature pyramid with strong semantics at every…
GLM
GLM, short for General Language Model, is both a pretraining framework for language understanding and generation and the name of a model family developed by researchers associated with Tsinghua University and…
Graph Machine Learning Models
Graph machine learning models are neural networks designed to operate on data structured as graphs, where the input is a set of nodes connected by edges rather than a grid like an image or a sequence like text.
Hidden Markov Model
A Hidden Markov Model (HMM) is a statistical model of sequential data in which an unobserved (hidden) sequence of discrete states follows a Markov process
Hyena
Hyena is a sub-quadratic, attention-free neural sequence operator that replaces the self-attention operator of the Transformer with a recurrence of long, implicitly parameterized convolutions and…
Infini-Attention
Infini-attention is an attention mechanism introduced by Google researchers Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal in the April 2024 paper "Leave No Context Behind: Efficient Infinite…
Jamba
Jamba is a family of open-weight large language models from AI21 Labs, first released on March 28, 2024, and is the world's first production-grade language model built on a Mamba state space model (SSM)…
Jamba2
Jamba2 is the second generation of hybrid State Space Model and Transformer language models released by AI21 Labs on January 8, 2026.
Joint Embedding Predictive Architecture
Joint Embedding Predictive Architecture (JEPA) is a family of self-supervised, non-generative neural network architectures proposed by Yann LeCun in his June 2022 position paper A Path Towards Autonomous…
Kimi Linear
Kimi Linear is a hybrid linear attention architecture published by Moonshot AI on October 30, 2025, together with a 48-billion-parameter mixture-of-experts model that activates 3 billion parameters per token.
Large Concept Model
A Large Concept Model (LCM) is a research approach to language modeling, introduced by Meta AI's Fundamental AI Research (FAIR) group in December 2024
Layer normalization
Layer normalization is a technique for normalizing the activations of a neural network across the feature dimension of each individual sample, rather than across a batch of samples.
Linear Attention
Linear attention is a family of sub-quadratic attention mechanisms that replaces the softmax dot-product operation of standard Transformer self-attention with a feature-map-based factorization
Liquid AI
Liquid AI is an artificial intelligence company founded in 2023 by researchers from MIT CSAIL and headquartered at 314 Main Street in Cambridge, Massachusetts.
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences.
Long-context language models
Long-context language models are large language models engineered to accept and reason over inputs far larger than the few-thousand-token windows used by early transformer systems, with frontier models in 2026…
LongNet
LongNet is a transformer variant introduced by Microsoft Research in July 2023 that is designed to scale attention to sequences exceeding one billion tokens while preserving performance on shorter inputs.
LongRoPE
LongRoPE is a context-window extension technique for large language models (LLMs) that use rotary position embeddings (RoPE).
MEGABYTE
MEGABYTE is a transformer architecture for autoregressive modeling of very long sequences directly at the byte level, introduced by researchers at Meta AI (FAIR) in May 2023.
MMDiT (Multimodal Diffusion Transformer)
MMDiT (Multimodal Diffusion Transformer, sometimes written MM-DiT) is a transformer architecture for text-conditioned image generation that gives image tokens and text tokens their own separate weights but…
Machine learning terms/Sequence Models
Sequence models are a class of machine learning systems designed to process inputs or produce outputs that have a meaningful order.
Mamba
Mamba is a neural network architecture for sequence modeling that uses selective state space models (SSMs) to process sequential data in linear time with respect to sequence length
Mamba 2
Mamba 2 is a state space model architecture introduced in the paper "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" by Tri Dao and Albert Gu
Mamba-3
Mamba-3 is a sequence-modeling architecture in the state space model (SSM) family, introduced in March 2026 by researchers at Carnegie Mellon University and Princeton.
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-Query Attention (MQA)
Multi-Query Attention (MQA) is a variant of the multi-head attention mechanism used in transformer neural networks in which all query heads share a single key head and a single value head, rather than each…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Multilayer Perceptron
A multilayer perceptron (MLP) is a feedforward neural network built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer
Node (neural network)
A node in a neural network is the basic computational element, an artificial neuron, that receives one or more inputs, multiplies each by a learned weight, sums them, adds a bias, and passes the result through…
PagedAttention
PagedAttention is a KV-cache memory management algorithm for serving large language models that applies the virtual-memory paging technique used by operating systems to the GPU, eliminating memory…
Perceiver
Perceiver is a family of general-purpose neural network architectures from DeepMind built around attention and a small latent bottleneck.
RMSNorm
RMSNorm (Root Mean Square Layer Normalization) is a feature normalization technique introduced by Biao Zhang and Rico Sennrich in 2019 that scales each activation vector by its root mean square only, dropping…
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
RadixAttention
RadixAttention is a KV cache management technique introduced in SGLang that uses a radix tree data structure to automatically share and reuse cached key-value tensors across inference requests.
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sliding window attention
Sliding window attention (SWA) is a sparse attention pattern in which each query token attends only to a fixed-size window of nearby tokens instead of to every preceding (or every other) token in the sequence.
SparDA
SparDA (Sparse Decoupled Attention) is an add-on architecture for long-context large language model inference proposed by researchers at NVIDIA in a paper posted to arXiv on 3 June 2026.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
SubQ
SubQ is a large language model released on May 5, 2026 by Subquadratic, a Miami-based startup that emerged from stealth claiming to have built the first frontier model on a fully subquadratic attention…
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.
Titans (neural architecture)
Titans is a family of neural sequence-modeling architectures from Google Research that combines an attention-based "short-term memory" with a deep neural "long-term memory" module that updates its own…
Tower
In deep learning, a tower is a self-contained sub-network inside a larger model that encodes one specific input or feature group into an output, usually an embedding
Transformers
A Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…
Unidirectional
Unidirectional is a property of a sequence model in which the representation or output at each position depends only on inputs from one direction of the sequence