Absolute Zero Reasoner
Absolute Zero Reasoner (AZR) is a research system for post-training a pretrained language model through self-play on machine-generated Python reasoning tasks.
Deep Learning
Explore Deep Learning through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Machine Learning
Articles that also belong to these categories. Counts cover all of Deep Learning.
All articles (333)Machine Learning (187)Neural Networks (81)Computer Vision (53)Training & Optimization (41)Natural Language Processing (37)Generative AI (31)Model Architecture (21)Transformer Models (18)Artificial Intelligence (16)Large Language Models (12)People (12)Reinforcement Learning (12)Developer Tools (11)Mathematics (11)AI Inference (9)Interpretability (8)AI Infrastructure (7)Speech & Audio AI (7)AI Hardware (6)AI Companies (5)AI History (5)Google DeepMind (5)Open Source AI (5)AI Research (4)AI for Science (4)Algorithms (4)Model Evaluation (4)Robotics (4)AI Models (3)Google (3)Meta AI (3)OpenAI (3)AI Alignment (2)AI Tools & Products (2)Chinese AI (2)Data & Datasets (2)Healthcare AI (2)MLOps (2)Microsoft (2)Multimodal AI (2)NVIDIA (2)Software Development (2)Statistics (2)AI Benchmarks (1)AI Ethics (1)AI Safety (1)Data Science (1)Diffusion Models (1)Embodied AI (1)Image Generation (1)Information Retrieval (1)Programming Languages (1)Prompt Engineering (1)Video Generation (1)
Showing 1-60 of 187 articles
Activation Function
An activation function is a nonlinear mathematical function applied to the output of each neuron in an artificial neural network, and it is what gives the network the ability to learn complex
Actor-critic methods
Actor-critic methods are a family of reinforcement learning algorithms that learn two things at once: a parameterized policy, called the actor, which chooses actions, and a value function, called the critic
AlexNet
AlexNet is a deep learning convolutional neural network, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto, that won the ImageNet Large Scale Visual Recognition…
Attention
Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them.
Attention Is All You Need
"Attention Is All You Need" is the 2017 research paper that introduced the transformer, the neural network architecture that underpins virtually every modern large language model.
Audio Classification Models
Audio classification models are machine learning systems for audio classification, the task of assigning one or more labels to an audio recording or to short fragments of it.
Backpropagation
Backpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation.
Batch
A batch in machine learning is the set of training examples processed together in one forward and backward pass before the model's parameters are updated once.
Batch Normalization
Batch normalization (BatchNorm or BN) is a normalization method used inside some neural networks.
Batch Size
In machine learning, batch size is the hyperparameter that sets how many training examples a model processes together before it updates its parameters with one step of gradient descent.
Bayesian Neural Network
A Bayesian neural network (BNN) is a neural network in which the weights and biases are represented as probability distributions rather than fixed point estimates.
Broadcasting
Broadcasting is the set of rules that lets element-wise operations (addition, subtraction, multiplication, division) act on arrays or tensors of different but compatible shapes by virtually stretching the…
CLIP (Contrastive Language-Image Pre-training)
CLIP, short for Contrastive Language-Image Pre-training, is a family of neural networks developed by OpenAI researchers to learn a shared representation of images and natural-language descriptions.
Calibration Layer
A calibration layer is a post-prediction adjustment appended to a trained machine learning model that rescales its raw output scores or predicted probabilities so they better reflect the true likelihood of…
Causal Language Model
A causal language model (CLM), also called an autoregressive language model or a decoder-only language model, is a language model that predicts the next token in a sequence using only the tokens that precede…
Chain of Thought Monitorability
Chain of thought monitorability is the property that lets safety researchers read a reasoning model's chain-of-thought (CoT), the step-by-step working it writes out in human language before answering, and…
Chain-of-Thought
Chain-of-thought (CoT) is a sequence of intermediate steps that a language model generates before its final answer.
Checkpoint
In machine learning, a checkpoint is a saved snapshot of a model's state captured at a specific point during the training process, used to resume training, recover from failures, or deploy and fine-tune the…
Chinchilla scaling laws
The Chinchilla scaling laws are a set of empirical findings published by DeepMind researchers in 2022 showing that, for a fixed compute budget, a large language model trains most efficiently when its number of…
Clipping
Clipping is a family of techniques in machine learning that constrain numerical values to lie within a specified range or below a specified magnitude.
Co-Adaptation
Co-adaptation in neural networks refers to a phenomenon in which different hidden units develop highly correlated behavior, becoming excessively dependent on one another rather than learning independent
Computer Vision
Computer vision is the study of computational methods that extract, estimate, or generate useful representations from visual measurements.
Connectionist Temporal Classification
Connectionist temporal classification (CTC) is a loss function and output layer design for training neural networks to label unsegmented sequences, such as transcribing an audio recording into characters when…
Context window
A context window is the finite token sequence that a language model can process for one invocation.
Continual learning
Continual learning, also called lifelong learning or incremental learning, is a machine learning paradigm in which a model learns from a stream of tasks or data distributions over time
Contrastive Learning
Contrastive learning is a family of machine learning methods that learn representations by pulling similar (positive) pairs of data points closer together in an embedding space while pushing dissimilar…
Convolution
Convolution is a mathematical operation that combines two functions to produce a third function, expressing how the shape of one is modified by the other.
Convolutional Filter
A convolutional filter (also called a kernel or feature detector) is a small matrix of learnable weights that slides across an input and computes a dot product at each position to produce a feature map.
Convolutional Layer
A convolutional layer is the core building block of a convolutional neural network (CNN): it slides a small set of learnable filters (also called kernels) across the input, computing a convolution (technically…
Convolutional Neural Network
A convolutional neural network (CNN, or ConvNet) is a neural network that uses convolution-like linear operators in at least some layers.
Critic
A critic in reinforcement learning (RL) is the component of an actor-critic system that estimates a value function, scoring how good the actor's chosen actions are and converting that judgment into a…
Cross-Entropy
Cross-entropy is a measure from information theory of how many bits (or nats) are needed to encode data drawn from a true probability distribution P when using a code optimized for a different
Cross-Entropy Loss
Cross-entropy loss is the standard loss function for classification and language modeling, defined as the negative log-probability a model assigns to the correct answer: for a single example its value is…
Curriculum learning
Curriculum learning is a training strategy for machine learning models in which training examples are presented in a meaningful, easy-to-hard order rather than at random
DDPM
Denoising Diffusion Probabilistic Models (DDPM) are a class of generative model introduced by Jonathan Ho, Ajay Jain, and Pieter Abbeel of UC Berkeley in their June 2020 paper "Denoising Diffusion…
DINO (computer vision)
DINO (self-DIstillation with NO labels) is a family of self-supervised learning methods for computer vision from Meta AI that trains Vision Transformers (ViTs) on unlabeled images and produces general-purpose…
Data Augmentation
Data augmentation is a set of techniques that artificially expand the size and diversity of a training dataset by applying label-preserving transformations to existing examples, rather than collecting new…
Data Parallelism
Data parallelism is a distributed training technique in which the same neural network model is replicated across multiple processing units (typically GPUs), each device trains on a different shard of the input…
Dataset API (tf.data)
The Dataset API (tf.data) is the high-performance input pipeline framework within TensorFlow for loading, transforming, and delivering data to machine learning models during training and evaluation.
Decoder
A decoder is the component of a neural network that turns an internal, compressed, or abstract representation into a desired output, such as a translated sentence, a generated image, a reconstructed input, or…
Deep Learning
Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to automatically learn representations of data at multiple levels of abstraction
Deep Model
A deep model, also called a deep learning model or deep neural network, is an artificial neural network built from many stacked layers of processing units that automatically learns hierarchical representations…
Deep Neural Network
A deep neural network (DNN) is an artificial neural network with multiple hidden layers of artificial neurons stacked between its input and output layers
Deep Q-Network (DQN)
Deep Q-Network (DQN) is a reinforcement learning algorithm that uses a deep neural network to approximate the optimal action-value function (Q-function)
DeepLIFT
DeepLIFT (Deep Learning Important FeaTures) is a feature attribution method for deep neural networks introduced by Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje at Stanford University in 2017 .
DeepSpeed
DeepSpeed is an open-source deep learning optimization library, originally developed by Microsoft, that makes distributed training and inference of large models efficient, easy to use, and cost-effective.
Denoising
Denoising is the process of removing unwanted noise from data to recover a cleaner underlying signal
Dense Layer
A dense layer, also called a fully connected (FC) layer, linear layer, or affine layer, is a layer in an artificial neural network where every input neuron is connected to every output neuron.
Diederik Kingma
Diederik Kingma is a Dutch machine learning researcher and a founding member of OpenAI who is best known as the first author of the Adam optimizer and the variational autoencoder (VAE)
Diffusion model
A diffusion model is a generative model that learns to transform samples from a simple reference distribution into samples resembling a data distribution by reversing a gradual corruption process.
Direct Preference Optimization (DPO)
Direct Preference Optimization (DPO) is a method for aligning large language models with human preferences that replaces the multi-stage reinforcement learning from human feedback (RLHF) pipeline with a single…
Double Descent
Double descent is a phenomenon in machine learning and statistical learning theory in which a model's test error, plotted against increasing model complexity, first traces the classical U-shaped bias-variance…
Downsampling
Downsampling is the process of reducing the number of samples, the spatial resolution, or the number of data instances in a signal, image, or dataset in order to lower computational cost and memory use while…
Dropout Regularization
Dropout regularization is a regularization technique for neural networks that prevents overfitting by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to…
Eager Execution
Eager execution is an imperative, define-by-run mode of running machine learning framework operations in which each operation is evaluated immediately as it is called and returns a concrete value
Early Stopping
Early stopping is a regularization technique that halts the training of an iterative machine learning model as soon as its performance on a held-out validation set stops improving
Embedding Space
An embedding space is a continuous, typically high-dimensional vector space in which data objects (words, sentences, images, users, audio clips, code, or other entities) are represented as dense numerical…
Embeddings
In machine learning, an embedding is a learned representation that maps an input into a vector, usually so that a downstream model or comparison rule can use relationships encoded in the vector's geometry.
Epoch
An epoch in machine learning is one complete pass through the entire training dataset, during which every example is presented to the model exactly once to compute gradients and update parameters before the…