ALBERT
ALBERT (A Lite BERT) is a parameter-efficient variant of the BERT language model developed by researchers at Google Research and the Toyota Technological Institute at Chicago (TTIC).
Transformer Models
Explore Transformer Models through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Deep Learning
Articles that also belong to these categories. Counts cover all of Transformer Models.
All articles (46)Deep Learning (18)Model Architecture (16)Machine Learning (11)Large Language Models (9)Natural Language Processing (9)Computer Vision (5)Neural Networks (4)AI Models (3)Microsoft (3)Mixture of Experts (3)Chinese AI (2)Google (2)Interpretability (2)Meta AI (2)Open Source AI (2)People (2)Training & Optimization (2)AI Companies (1)AI Hardware (1)AI Inference (1)Algorithms (1)Artificial Intelligence (1)Diffusion Models (1)Generative AI (1)Google DeepMind (1)Healthcare AI (1)Image Generation (1)Multimodal AI (1)Robotics (1)
Showing 1-18 of 18 articles
DETR
DETR (DEtection TRansformer) is an end-to-end object detection model that reframes detection as a direct set prediction problem solved with a transformer encoder-decoder and bipartite matching, removing the…
DeBERTa
DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a family of pre-trained language models developed by Microsoft Research that improves BERT and RoBERTa with two innovations: a disentangled…
DeiT
DeiT (Data-efficient Image Transformers) is a family of vision transformer models that proved Vision Transformers can be trained to state-of-the-art image classification accuracy on ImageNet alone
DistilBERT
DistilBERT is a compressed version of BERT released by Hugging Face in October 2019 that is 40% smaller and 60% faster than BERT-base while retaining 97% of its language-understanding performance on the GLUE…
ELECTRA
ELECTRA, which stands for Efficiently Learning an Encoder that Classifies Token Replacements Accurately
Grouped-Query Attention
Grouped-query attention (GQA) is an attention mechanism for transformer language models that partitions the query heads into a small number of groups, where every query head in a group shares one key…
KV Cache
KV cache, short for key-value cache, is transient model state used during Transformer generation.
Mixture of Depths
Mixture of Depths (MoD) is a technique for dynamically allocating computation to individual tokens within transformer-based language models.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Positional encoding
Positional encoding is a technique used to inject information about token order into transformer models.
RoBERTa
RoBERTa (Robustly Optimized BERT Pretraining Approach) is an open-source natural language processing model released in July 2019 by researchers at Facebook AI (now Meta AI) and the University of Washington…
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping windows and introduces a shifted window…
XLNet
XLNet is a generalized autoregressive pretraining method for natural language processing that combines the strengths of autoregressive and autoencoding language models.