Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Transformer Models
Explore Transformer Models through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Neural Networks
Articles that also belong to these categories. Counts cover all of Transformer Models.
All articles (46)Deep Learning (18)Model Architecture (16)Machine Learning (11)Large Language Models (9)Natural Language Processing (9)Computer Vision (5)Neural Networks (4)AI Models (3)Microsoft (3)Mixture of Experts (3)Chinese AI (2)Google (2)Interpretability (2)Meta AI (2)Open Source AI (2)People (2)Training & Optimization (2)AI Companies (1)AI Hardware (1)AI Inference (1)Algorithms (1)Artificial Intelligence (1)Diffusion Models (1)Generative AI (1)Google DeepMind (1)Healthcare AI (1)Image Generation (1)Multimodal AI (1)Robotics (1)
Showing 1-4 of 4 articles
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping windows and introduces a shifted window…