Induction Heads
Induction heads are a circuit pattern in Transformer language models in which a small set of attention heads, typically spread across two layers, perform an in-context "match and copy" operation that completes…
Transformer Models
Explore Transformer Models through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Interpretability
Articles that also belong to these categories. Counts cover all of Transformer Models.
All articles (46)Deep Learning (18)Model Architecture (16)Machine Learning (11)Large Language Models (9)Natural Language Processing (9)Computer Vision (5)Neural Networks (4)AI Models (3)Microsoft (3)Mixture of Experts (3)Chinese AI (2)Google (2)Interpretability (2)Meta AI (2)Open Source AI (2)People (2)Training & Optimization (2)AI Companies (1)AI Hardware (1)AI Inference (1)Algorithms (1)Artificial Intelligence (1)Diffusion Models (1)Generative AI (1)Google DeepMind (1)Healthcare AI (1)Image Generation (1)Multimodal AI (1)Robotics (1)
Showing 1-2 of 2 articles
Logit lens
The logit lens is a foundational technique in mechanistic interpretability for inspecting the intermediate computations of transformer language models.