Infini-Attention
Infini-attention is an attention mechanism introduced by Google researchers Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal in the April 2024 paper "Leave No Context Behind: Efficient Infinite…
Explore Google through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Transformer Models
Articles that also belong to these categories. Counts cover all of Google.
All articles (102)AI Hardware (24)Machine Learning (16)AI Tools & Products (14)AI Infrastructure (13)Large Language Models (12)AI Models (11)Developer Tools (11)Open Source AI (11)Artificial Intelligence (7)Generative AI (7)People (7)Google DeepMind (6)Multimodal AI (6)AI Agents (5)AI Code Generation (5)AI History (5)Conversational AI (5)Image Generation (5)AI Research (4)Healthcare AI (4)Speech & Audio AI (4)AI Companies (3)AI for Science (3)Computer Vision (3)Deep Learning (3)Enterprise AI (3)Music & Audio Generation (3)AI Inference (2)Data Centers (2)Model Architecture (2)Small Language Models (2)Software Development (2)Transformer Models (2)AI Safety (1)Algorithms (1)Anthropic (1)Autonomous Vehicles (1)Computer Science (1)Data & Datasets (1)Diffusion Models (1)Drug Discovery (1)Edge computing (1)Education AI (1)MLOps (1)Mathematics (1)Mixture of Experts (1)Neural Networks (1)Reinforcement Learning (1)Robotics (1)Training & Optimization (1)Video Generation (1)Voice AI (1)
Showing 1-2 of 2 articles
Switch Transformer
The Switch Transformer is a sparsely activated Mixture of Experts (MoE) Transformer architecture introduced by William Fedus, Barret Zoph, and Noam Shazeer at Google in January 2021.