Action Chunking with Transformers (ACT)
Action Chunking with Transformers (ACT) is an imitation learning algorithm for fine-grained robotic manipulation that predicts a short sequence (a "chunk") of future actions at once instead of a single next…
Machine Learning
Explore learning methods, model architectures, datasets, and practical applications.
Explore articles
Browse subtopics: Transformer Models
Articles that also belong to these categories. Counts cover all of Machine Learning.
All articles (858)Deep Learning (187)Natural Language Processing (92)Statistics (90)Training & Optimization (70)Data & Datasets (65)Neural Networks (58)Reinforcement Learning (58)Model Evaluation (55)Artificial Intelligence (54)Large Language Models (49)Mathematics (48)Computer Vision (41)AI Ethics (35)AI Infrastructure (29)AI Safety (26)Open Source AI (25)Data Science (24)AI Benchmarks (22)Developer Tools (20)Algorithms (18)AI Hardware (17)AI Tools & Products (17)Google (16)Generative AI (14)Information Retrieval (14)Model Architecture (13)People (13)AI Agents (12)Interpretability (12)Robotics (12)AI Companies (11)Transformer Models (11)AI Inference (10)MLOps (10)AI Models (8)Computer Science (8)AI Code Generation (7)AI Research (7)Multimodal AI (7)Speech & Audio AI (7)Chinese AI (6)AI Policy & Regulation (5)Meta AI (5)Programming Languages (5)AI History (4)AI for Science (4)Prompt Engineering (4)AI Alignment (3)AI Events (3)Reasoning Models (3)Embodied AI (2)Software Development (2)2025 in artificial intelligence (1)Anthropic (1)Diffusion Models (1)Drug Discovery (1)Education AI (1)Enterprise AI (1)Healthcare AI (1)Image Generation (1)NVIDIA (1)Small Language Models (1)World Models (1)
Showing 1-11 of 11 articles
Grouped-Query Attention
Grouped-query attention (GQA) is an attention mechanism for transformer language models that partitions the query heads into a small number of groups, where every query head in a group shares one key…
KV Cache
KV cache, short for key-value cache, is transient model state used during Transformer generation.
Masked autoencoder (MAE)
Masked autoencoder (MAE) is a self-supervised learning method for vision transformers that masks roughly 75% of an input image's patches and trains a network to reconstruct the missing pixels from the small…
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
RoBERTa
RoBERTa (Robustly Optimized BERT Pretraining Approach) is an open-source natural language processing model released in July 2019 by researchers at Facebook AI (now Meta AI) and the University of Washington…
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
XLNet
XLNet is a generalized autoregressive pretraining method for natural language processing that combines the strengths of autoregressive and autoencoding language models.