Showing 1-44 of 44 articles
ALBERT
ALBERT (A Lite BERT) is a parameter-efficient variant of the BERT language model developed by researchers at Google Research and the Toyota Technological...
Deep LearningNatural Language Processing
ALiBi (Attention with Linear Biases)
ALiBi (Attention with Linear Biases) is a positional encoding method for transformer language models that, instead of adding positional embeddings to word...
Model Architecture
Action Chunking with Transformers (ACT)
Action Chunking with Transformers (ACT) is an imitation learning algorithm for fine-grained robotic manipulation that predicts a short sequence (a "chunk") of...
Machine LearningRobotics
Aidan Gomez
Aidan N. Gomez (born 1996) is a British-Canadian computer scientist and technology executive who is the co-founder and chief executive officer of Cohere, a...
AI CompaniesPeople
Ashish Vaswani
Ashish Vaswani (born 1986) is an Indian-American computer scientist who is the first-listed author of the 2017 paper "Attention Is All You Need," the work that...
People
BERT
BERT (Bidirectional Encoder Representations from Transformers) is a transformer-based encoder-only language model developed by researchers at Google AI...
Large Language Models
BioBERT
BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) is a domain-specific language model that adapts BERT to...
Healthcare AILarge Language Models
Cross-attention
Cross-attention is a variant of the attention mechanism in which the queries are derived from one sequence or representation while the keys and values are...
Model Architecture
DETR
DETR (DEtection TRansformer) is an end-to-end object detection model that reframes detection as a direct set prediction problem solved with a transformer...
Computer VisionDeep Learning
DeBERTa
DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a family of pre-trained language models developed by Microsoft Research that improves BERT and...
Deep LearningMicrosoft
DeiT
DeiT (Data-efficient Image Transformers) is a family of vision transformer models that proved Vision Transformers can be trained to state-of-the-art image...
Computer VisionDeep Learning
Differential Transformer
The Differential Transformer (often shortened to Diff Transformer or DIFF Transformer) is a decoder-only neural sequence architecture introduced by researchers...
MicrosoftModel Architecture
Diffusion Transformer (DiT)
A Diffusion Transformer (DiT) is a transformer-based neural network backbone for diffusion models that replaces the U-Net with a Vision Transformer operating...
Diffusion ModelsGenerative AI
DistilBERT
DistilBERT is a compressed version of BERT released by Hugging Face in October 2019 that is 40% smaller and 60% faster than BERT-base while retaining 97% of...
AI ModelsDeep Learning
ELECTRA
ELECTRA, which stands for Efficiently Learning an Encoder that Classifies Token Replacements Accurately, is a pre-training method for natural language...
Deep LearningNatural Language Processing
Flash Attention 3
Flash Attention 3 (FA3, styled FlashAttention-3) is the third generation of the FlashAttention algorithm, an exact-attention GPU kernel designed to exploit...
AI HardwareAlgorithms
Grouped-Query Attention
Grouped-query attention (GQA) is an attention mechanism for transformer language models that partitions the query heads into a small number of groups, where...
Deep LearningMachine Learning
Hiera
Hiera is a hierarchical vision transformer from Meta AI (FAIR), introduced in the paper "Hiera: A Hierarchical Vision Transformer without the...
Computer VisionMeta AI
Induction Heads
Induction heads are a circuit pattern in Transformer language models in which a small set of attention heads, typically spread across two layers, perform an...
Interpretability
Infini-Attention
Infini-attention is an attention mechanism introduced by Google researchers Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal in the April 2024 paper...
GoogleModel Architecture
KV Cache
A KV cache (key-value cache) is a memory optimization technique used during transformer inference that stores previously computed key and value tensors from...
AI InferenceDeep Learning
Linear Attention
Linear attention is a family of sub-quadratic attention mechanisms that replaces the softmax dot-product operation of standard Transformer self-attention with...
Model Architecture
Logit lens
The logit lens is a foundational technique in mechanistic interpretability for inspecting the intermediate computations of transformer language models. It...
Interpretability
LongNet
LongNet is a transformer variant introduced by Microsoft Research in July 2023 that is designed to scale attention to sequences exceeding one billion tokens...
MicrosoftModel Architecture
Longformer
Longformer is a transformer architecture for processing long documents, introduced by Iz Beltagy, Matthew E. Peters, and Arman Cohan of the Allen Institute for...
Large Language ModelsNatural Language Processing
MEGABYTE
MEGABYTE is a transformer architecture for autoregressive modeling of very long sequences directly at the byte level, introduced by researchers at Meta AI...
Meta AIModel Architecture
Masked autoencoder (MAE)
Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network...
Computer VisionMachine Learning
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models. Introduced by...
Deep LearningMachine Learning
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations...
Deep LearningMachine Learning
Multi-Query Attention (MQA)
Multi-Query Attention (MQA) is a variant of the multi-head attention mechanism used in transformer neural networks in which all query heads share a single key...
Model Architecture
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or...
Deep LearningMachine Learning
PaLM
PaLM (Pathways Language Model) is a family of large language models developed by Google Research. The original PaLM, announced on April 4, 2022, was a...
Google DeepMindLarge Language Models
Positional encoding
Positional encoding is a technique used to inject information about token order into transformer models. Because transformers process all tokens in a sequence...
Deep LearningNatural Language Processing
RMSNorm
RMSNorm (Root Mean Square Layer Normalization) is a feature normalization technique introduced by Biao Zhang and Rico Sennrich in 2019 that scales each...
Artificial IntelligenceModel Architecture
Ring Attention
Ring Attention, formally Ring Attention with Blockwise Transformers, is a distributed algorithm for computing the self-attention operation of transformer...
Training & Optimization
RoBERTa
RoBERTa (Robustly Optimized BERT Pretraining Approach) is an open-source natural language processing model released in July 2019 by researchers at Facebook AI...
Deep LearningMachine Learning
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key...
Deep LearningLarge Language Models
Self-attention
Self-attention is a mechanism that lets a neural network weigh how much every element of a single input sequence should influence every other element,...
Deep LearningMachine Learning
Sliding window attention
Sliding window attention (SWA) is a sparse attention pattern in which each query token attends only to a fixed-size window of nearby tokens instead of to every...
Model Architecture
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token...
Deep LearningMachine Learning
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping...
Computer VisionDeep Learning
Switch Transformer
The Switch Transformer is a sparsely activated Mixture of Experts (MoE) Transformer architecture introduced by William Fedus, Barret Zoph, and Noam Shazeer at...
GoogleLarge Language Models
T5 (language model)
T5 (Text-to-Text Transfer Transformer) is a family of transformer-based language models released by Google in 2019-2020 that reframes every natural language...
Large Language Models
XLNet
XLNet is a generalized autoregressive pretraining method for natural language processing that combines the strengths of autoregressive and autoencoding...
Deep LearningMachine Learning