ALBERT
ALBERT (A Lite BERT) is a parameter-efficient variant of the BERT language model developed by researchers at Google Research and the Toyota Technological Institute at Chicago (TTIC).
Deep Learning
Explore Deep Learning through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Natural Language Processing
Articles that also belong to these categories. Counts cover all of Deep Learning.
All articles (333)Machine Learning (187)Neural Networks (81)Computer Vision (53)Training & Optimization (41)Natural Language Processing (37)Generative AI (31)Model Architecture (21)Transformer Models (18)Artificial Intelligence (16)Large Language Models (12)People (12)Reinforcement Learning (12)Developer Tools (11)Mathematics (11)AI Inference (9)Interpretability (8)AI Infrastructure (7)Speech & Audio AI (7)AI Hardware (6)AI Companies (5)AI History (5)Google DeepMind (5)Open Source AI (5)AI Research (4)AI for Science (4)Algorithms (4)Model Evaluation (4)Robotics (4)AI Models (3)Google (3)Meta AI (3)OpenAI (3)AI Alignment (2)AI Tools & Products (2)Chinese AI (2)Data & Datasets (2)Healthcare AI (2)MLOps (2)Microsoft (2)Multimodal AI (2)NVIDIA (2)Software Development (2)Statistics (2)AI Benchmarks (1)AI Ethics (1)AI Safety (1)Data Science (1)Diffusion Models (1)Embodied AI (1)Image Generation (1)Information Retrieval (1)Programming Languages (1)Prompt Engineering (1)Video Generation (1)
Showing 1-37 of 37 articles
Attention Is All You Need
"Attention Is All You Need" is the 2017 research paper that introduced the transformer, the neural network architecture that underpins virtually every modern large language model.
Bahdanau attention
Bahdanau attention is the first attention mechanism for neural networks, introduced in 2014 to let a sequence-to-sequence decoder soft-align to every encoder hidden state instead of relying on a single…
Causal Language Model
A causal language model (CLM), also called an autoregressive language model or a decoder-only language model, is a language model that predicts the next token in a sequence using only the tokens that precede…
Chain-of-Thought
Chain-of-thought (CoT) is a sequence of intermediate steps that a language model generates before its final answer.
Context window
A context window is the finite token sequence that a language model can process for one invocation.
DeBERTa
DeBERTa (Decoding-enhanced BERT with Disentangled Attention) is a family of pre-trained language models developed by Microsoft Research that improves BERT and RoBERTa with two innovations: a disentangled…
Direct Preference Optimization (DPO)
Direct Preference Optimization (DPO) is a method for aligning large language models with human preferences that replaces the multi-stage reinforcement learning from human feedback (RLHF) pipeline with a single…
DistilBERT
DistilBERT is a compressed version of BERT released by Hugging Face in October 2019 that is 40% smaller and 60% faster than BERT-base while retaining 97% of its language-understanding performance on the GLUE…
ELECTRA
ELECTRA, which stands for Efficiently Learning an Encoder that Classifies Token Replacements Accurately
Embeddings
In machine learning, an embedding is a learned representation that maps an input into a vector, usually so that a downstream model or comparison rule can use relationships encoded in the vector's geometry.
Full Softmax
Full softmax (also called the standard softmax or exact softmax) is the softmax computation that calculates a probability for every possible output class in a classification model
In-Context Learning
In-context learning (ICL) is the ability of a large language model to learn a new task at inference time by conditioning on a prompt that contains a few input-output examples (demonstrations)
Instruction Tuning
Instruction tuning is the post-pretraining training stage in which a large language model (LLM) is fine-tuned on a curated collection of (instruction
Language Model
A language model is a model of patterns in language that assigns probabilities or comparable scores to linguistic sequences. Its units may be characters, words, subwords, bytes, or other tokens.
Large Language Model
A large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…
LoRA (Low-Rank Adaptation)
Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.
Masked Language Model
A masked language model (MLM) is a language model trained to predict missing tokens that have been hidden in a sequence of text, using context from both the left and the right of each blank.
Multimodal Model
A multimodal model is a machine learning model, or a model-based system, that processes, relates, or produces information across more than one kind of data. Each kind is called a modality.
Neural Machine Translation
Neural machine translation (NMT) is an approach to machine translation in which a single artificial neural network, trained end to end on bilingual text
OCR Models
OCR Models are artificial intelligence (AI) systems that convert images of typed, handwritten, or printed text into machine-readable digital text through Optical Character Recognition (OCR).
Positional encoding
Positional encoding is a technique used to inject information about token order into transformer models.
Post-training
Post-training is the stage of large language model (LLM) development that comes after pre-training and turns a raw, general-purpose base model into an aligned, instruction-following AI assistant.
Pre-Trained Model
A pre-trained model is a machine learning model that has already been trained on a large, general-purpose dataset and can then be reused, either as a fixed feature extractor or by fine-tuning
Pre-training
Pre-training is a stage of machine learning in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use.
Reinforcement Learning from Human Feedback (RLHF)
Reinforcement learning from human feedback (RLHF) is a family of methods in which judgments supplied by people are converted into a reward signal and a policy is optimized with reinforcement learning against…
RoBERTa
RoBERTa (Robustly Optimized BERT Pretraining Approach) is an open-source natural language processing model released in July 2019 by researchers at Facebook AI (now Meta AI) and the University of Washington…
Sequence Model
A sequence model is a class of machine learning models that processes, generates, or predicts ordered data where the position and surrounding context of each element carry meaning.
Sequence-to-Sequence Task
A sequence-to-sequence (seq2seq) task is any machine learning problem in which a model receives a variable-length input sequence and produces a variable-length output sequence, where the input and output may…
Speech Recognition
Speech recognition, usually called automatic speech recognition (ASR), is the computational task of converting a spoken-language signal into a sequence of written symbols.
Supervised fine-tuning
Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.
Text summarization
Text summarization is the natural language processing (NLP) task of automatically producing a shorter version of one or more documents that preserves the most important information from the original text.
Token
A token is the basic unit of text that a language model reads and writes: a word, a subword fragment, a single character, or a byte, produced by splitting text during a step called tokenization.
Wav2Vec
Wav2Vec is a family of self-supervised learning models from Meta AI (formerly Facebook AI Research) that learn speech representations directly from raw audio waveforms
Whisper
Whisper is an open-source family of automatic speech recognition (ASR) models developed by OpenAI and first released on September 21, 2022.
Word Embedding
A word embedding is a learned representation of text in which words are mapped to dense vectors of real numbers in a continuous vector space, so that words with similar meanings are positioned close together.
XLNet
XLNet is a generalized autoregressive pretraining method for natural language processing that combines the strengths of autoregressive and autoencoding language models.