ALBERT
ALBERT (A Lite BERT) is a parameter-efficient variant of the BERT language model developed by researchers at Google Research and the Toyota Technological Institute at Chicago (TTIC).
Transformer Models
Explore Transformer Models through related topics and the articles other pages reference most.
Most referenced in this topic
Ranked by links from other AI Wiki pages.
Explore articles
Browse subtopics (29)
Articles that also belong to these categories. Counts cover all of Transformer Models.
All articles (46)Deep Learning (18)Model Architecture (16)Machine Learning (11)Large Language Models (9)Natural Language Processing (9)Computer Vision (5)Neural Networks (4)AI Models (3)Microsoft (3)Mixture of Experts (3)Chinese AI (2)Google (2)Interpretability (2)Meta AI (2)Open Source AI (2)People (2)Training & Optimization (2)AI Companies (1)AI Hardware (1)AI Inference (1)Algorithms (1)Artificial Intelligence (1)Diffusion Models (1)Generative AI (1)Google DeepMind (1)Healthcare AI (1)Image Generation (1)Multimodal AI (1)Robotics (1)
Showing 1-46 of 46 articles
ALiBi (Attention with Linear Biases)
ALiBi (Attention with Linear Biases) is a positional encoding method for transformer language models that, instead of adding positional embeddings to word vectors, adds a static, non-learned penalty to…
Action Chunking with Transformers (ACT)
Action Chunking with Transformers (ACT) is an imitation learning algorithm for fine-grained robotic manipulation that predicts a short sequence (a "chunk") of future actions at once instead of a single next…
Aidan Gomez
Aidan N. Gomez (born 1996) is a British-Canadian computer scientist and technology executive who is the co-founder and chief executive officer of Cohere
Ashish Vaswani
Ashish Vaswani (born 1986) is an Indian-American computer scientist who is the first-listed author of the 2017 paper "Attention Is All You Need," the work that introduced the transformer architecture now…
BERT
BERT, short for Bidirectional Encoder Representations from Transformers, is a pretrained language model introduced by researchers at Google Research in 2018.
BioBERT
BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) is a domain-specific language model that adapts BERT to biomedicine by continuing its pre-training on large…
Cross-attention
Cross-attention is a variant of the attention mechanism in which the queries are derived from one sequence or representation while the keys and values are derived from a different sequence or representation
DETR
DETR (DEtection TRansformer) is an end-to-end object detection model that reframes detection as a direct set prediction problem solved with a transformer encoder-decoder and bipartite matching, removing the…
DeBERTa
DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a family of pre-trained language models developed by Microsoft Research that improves BERT and RoBERTa with two innovations: a disentangled…
DeiT
DeiT (Data-efficient Image Transformers) is a family of vision transformer models that proved Vision Transformers can be trained to state-of-the-art image classification accuracy on ImageNet alone
Differential Transformer
The Differential Transformer (often shortened to Diff Transformer or DIFF Transformer) is a decoder-only neural sequence architecture introduced by researchers at Microsoft Research and Tsinghua University in…
Diffusion Transformer (DiT)
A Diffusion Transformer (DiT) is a transformer-based neural network backbone for diffusion models that replaces the U-Net with a Vision Transformer operating on patches of an image latent.
DistilBERT
DistilBERT is a compressed version of BERT released by Hugging Face in October 2019 that is 40% smaller and 60% faster than BERT-base while retaining 97% of its language-understanding performance on the GLUE…
ELECTRA
ELECTRA, which stands for Efficiently Learning an Encoder that Classifies Token Replacements Accurately
Flash Attention 3
Flash Attention 3 (FA3, styled FlashAttention-3) is the third generation of the FlashAttention algorithm
Grouped-Query Attention
Grouped-query attention (GQA) is an attention mechanism for transformer language models that partitions the query heads into a small number of groups, where every query head in a group shares one key…
Hiera
Hiera is a hierarchical vision transformer from Meta AI (FAIR), introduced in the paper "Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles" presented as an oral at the International…
Hy4 Preview
Hy4 Preview is an open-weight large language model released by the Tencent Hy Team on August 28, 2026.
Induction Heads
Induction heads are a circuit pattern in Transformer language models in which a small set of attention heads, typically spread across two layers, perform an in-context "match and copy" operation that completes…
Infini-Attention
Infini-attention is an attention mechanism introduced by Google researchers Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal in the April 2024 paper "Leave No Context Behind: Efficient Infinite…
KV Cache
KV cache, short for key-value cache, is transient model state used during Transformer generation.
Linear Attention
Linear attention is a family of sub-quadratic attention mechanisms that replaces the softmax dot-product operation of standard Transformer self-attention with a feature-map-based factorization
Logit lens
The logit lens is a foundational technique in mechanistic interpretability for inspecting the intermediate computations of transformer language models.
LongNet
LongNet is a transformer variant introduced by Microsoft Research in July 2023 that is designed to scale attention to sequences exceeding one billion tokens while preserving performance on shorter inputs.
Longformer
Longformer is a transformer architecture for processing long documents, introduced by Iz Beltagy, Matthew E. Peters
MEGABYTE
MEGABYTE is a transformer architecture for autoregressive modeling of very long sequences directly at the byte level, introduced by researchers at Meta AI (FAIR) in May 2023.
Masked autoencoder (MAE)
Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network to reconstruct the missing pixels from the small…
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-Query Attention (MQA)
Multi-Query Attention (MQA) is a variant of the multi-head attention mechanism used in transformer neural networks in which all query heads share a single key head and a single value head, rather than each…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
PaLM
PaLM (Pathways Language Model) is a family of dense, decoder-only large language models developed by Google Research. Google announced the original family on April 4, 2022.
Positional encoding
Positional encoding is a technique used to inject information about token order into transformer models.
Qwen3.8-Flash-Next
Qwen3.8-Flash-Next is an experimental open-weight multimodal large language model released by Alibaba Group's Qwen team on August 26, 2026.
RMSNorm
RMSNorm (Root Mean Square Layer Normalization) is a feature normalization technique introduced by Biao Zhang and Rico Sennrich in 2019 that scales each activation vector by its root mean square only, dropping…
Ring Attention
Ring Attention, formally Ring Attention with Blockwise Transformers, is a distributed algorithm for computing the self-attention operation of transformer neural networks across a ring of compute devices
RoBERTa
RoBERTa (Robustly Optimized BERT Pretraining Approach) is an open-source natural language processing model released in July 2019 by researchers at Facebook AI (now Meta AI) and the University of Washington…
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sliding window attention
Sliding window attention (SWA) is a sparse attention pattern in which each query token attends only to a fixed-size window of nearby tokens instead of to every preceding (or every other) token in the sequence.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping windows and introduces a shifted window…
Switch Transformer
The Switch Transformer is a sparsely activated Mixture of Experts (MoE) Transformer architecture introduced by William Fedus, Barret Zoph, and Noam Shazeer at Google in January 2021.
T5 (language model)
T5 (Text-to-Text Transfer Transformer) is a family of transformer-based language models released by Google in 2019-2020 that reframes every natural language processing (NLP) task, classification, translation…
XLNet
XLNet is a generalized autoregressive pretraining method for natural language processing that combines the strengths of autoregressive and autoencoding language models.