Infini-Attention
Infini-attention is an attention mechanism introduced by Google researchers Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal in the April 2024 paper "Leave No Context Behind: Efficient Infinite…
Transformer Models
Explore Transformer Models through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Google
Articles that also belong to these categories. Counts cover all of Transformer Models.
All articles (46)Deep Learning (18)Model Architecture (16)Machine Learning (11)Large Language Models (9)Natural Language Processing (9)Computer Vision (5)Neural Networks (4)AI Models (3)Microsoft (3)Mixture of Experts (3)Chinese AI (2)Google (2)Interpretability (2)Meta AI (2)Open Source AI (2)People (2)Training & Optimization (2)AI Companies (1)AI Hardware (1)AI Inference (1)Algorithms (1)Artificial Intelligence (1)Diffusion Models (1)Generative AI (1)Google DeepMind (1)Healthcare AI (1)Image Generation (1)Multimodal AI (1)Robotics (1)
Showing 1-2 of 2 articles
Switch Transformer
The Switch Transformer is a sparsely activated Mixture of Experts (MoE) Transformer architecture introduced by William Fedus, Barret Zoph, and Noam Shazeer at Google in January 2021.