RNN
RNN is the standard abbreviation for recurrent neural network, a class of artificial neural network in which connections between units form cycles
Deep Learning
Explore Deep Learning through related topics and the articles other pages reference most.
Explore articles
Browse subtopics (54)
Articles that also belong to these categories. Counts cover all of Deep Learning.
All articles (333)Machine Learning (187)Neural Networks (81)Computer Vision (53)Training & Optimization (41)Natural Language Processing (37)Generative AI (31)Model Architecture (21)Transformer Models (18)Artificial Intelligence (16)Large Language Models (12)People (12)Reinforcement Learning (12)Developer Tools (11)Mathematics (11)AI Inference (9)Interpretability (8)AI Infrastructure (7)Speech & Audio AI (7)AI Hardware (6)AI Companies (5)AI History (5)Google DeepMind (5)Open Source AI (5)AI Research (4)AI for Science (4)Algorithms (4)Model Evaluation (4)Robotics (4)AI Models (3)Google (3)Meta AI (3)OpenAI (3)AI Alignment (2)AI Tools & Products (2)Chinese AI (2)Data & Datasets (2)Healthcare AI (2)MLOps (2)Microsoft (2)Multimodal AI (2)NVIDIA (2)Software Development (2)Statistics (2)AI Benchmarks (1)AI Ethics (1)AI Safety (1)Data Science (1)Diffusion Models (1)Embodied AI (1)Image Generation (1)Information Retrieval (1)Programming Languages (1)Prompt Engineering (1)Video Generation (1)
Showing 241-300 of 333 articles
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
ReLU
ReLU, short for rectified linear unit, is an activation function that maps a real input to its positive part:
Rectified Linear Unit (ReLU)
The Rectified Linear Unit (ReLU) is the most widely used activation function in deep learning, defined mathematically as $$f(x) = \max(0, x)$$: it returns the input directly when positive and outputs zero…
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Register tokens (Vision Transformers Need Registers)
Register tokens are a small set of extra learnable tokens added to the input sequence of a Vision Transformer (ViT) so the network has a dedicated place to carry out internal, image-level computation.
Regularization
Regularization is any deliberate change to a learning problem or training algorithm intended to improve performance on data that were not used to fit the model.
Reinforcement Learning from Human Feedback (RLHF)
Reinforcement learning from human feedback (RLHF) is a family of methods in which judgments supplied by people are converted into a reward signal and a policy is optimized with reinforcement learning against…
Reinforcement learning
Reinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by taking actions in an environment to maximize a cumulative reward signal, discovering good behavior…
Replay Buffer
A replay buffer (also called an experience replay buffer or replay memory) is a fixed-size memory that stores an off-policy reinforcement learning agent's past transitions, each a tuple of state, action…
Representation
A representation in machine learning is the format in which a data example is encoded before a model processes it: the same image can be represented as raw pixels, a list of hand-designed measurements, or a…
Representation learning
Representation learning is the area of machine learning concerned with getting a system to discover the features it needs for a task, instead of having people specify those features by hand.
ResNet
ResNet, short for residual network, is a family of deep convolutional neural networks introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at CVPR 2016.
Residual connection
A residual connection (also called a skip connection or shortcut connection) is a structural element in neural networks that adds the input of a layer or block directly to its output
RoBERTa
RoBERTa (Robustly Optimized BERT Pretraining Approach) is an open-source natural language processing model released in July 2019 by researchers at Facebook AI (now Meta AI) and the University of Washington…
Robot learning
Robot learning studies how robots acquire or improve behavior from data and experience.
Rotary Position Embedding
Rotary Position Embedding (RoPE) is a positional encoding method for transformer models that encodes a token's absolute position by rotating its query and key vectors in two-dimensional subspaces
SDEdit
SDEdit (Stochastic Differential Editing) is a method for guided image synthesis and editing that turns a rough user guide, such as a stroke painting, a coarse collage, or a real photograph with edits pasted…
SOAP (optimizer)
SOAP (ShampoO with Adam in the Preconditioner's eigenbasis) is a second-order optimization algorithm for training deep neural networks, introduced by Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai…
SSD (Single Shot MultiBox Detector)
SSD (Single Shot MultiBox Detector) is a single stage object detection model that predicts bounding boxes and per class confidence scores in one forward pass through a convolutional network
Saliency map
A saliency map is an explainable AI visualization that highlights which parts of an input, most often the individual pixels of an image, most influenced a deep learning model's prediction.
SambaNova Systems
SambaNova Systems is an American artificial intelligence hardware and software company headquartered in San Jose, California.
Scaling Laws
Neural scaling laws are empirical relationships that describe how a measured outcome changes as a neural network, its training data, or the computation used to train it becomes larger.
Schedule-Free optimizer
Schedule-Free is a family of optimization algorithms for deep learning and convex stochastic optimization that matches or exceeds the performance of tuned learning-rate schedules without specifying a horizon T…
Self-Extend
Self-Extend (written SelfExtend in the original paper) is a training-free technique that lets a pretrained large language model process inputs much longer than the context window it was trained on, with no…
Self-Supervised Learning
Self-supervised learning (SSL) is a machine learning approach in which a model learns representations from unlabeled data by generating its own supervisory signal from the structure of the data itself
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Self-training
Self-training is a semi-supervised learning procedure in which a model trained on a small labeled set is used to generate predictions on unlabeled data
Semantic Segmentation
Semantic segmentation is a computer vision task that assigns a category label to every single pixel in an image, producing a dense map in which each pixel carries the identity of the object class it belongs to.
Sequence Model
A sequence model is a class of machine learning models that processes, generates, or predicts ordered data where the position and surrounding context of each element carry meaning.
Sequence Parallelism
Sequence parallelism (SP) is a family of distributed training techniques for transformer-based neural networks that partitions activations along the sequence (token) dimension across multiple accelerators…
Sequence-to-Sequence Task
A sequence-to-sequence (seq2seq) task is any machine learning problem in which a model receives a variable-length input sequence and produces a variable-length output sequence, where the input and output may…
Shampoo (optimizer)
Shampoo is a second-order stochastic optimization algorithm for training neural networks that maintains, for each parameter tensor
Show-o
Show-o is a unified multimodal model, introduced in 2024, that handles both multimodal understanding and visual generation inside a single Transformer.
Sigmoid Function
The sigmoid function is a mathematical function that maps any real number to a value between 0 and 1 using the formula $$\sigma(x) = \frac{1}{1 + e^{-x}}$$, producing a smooth S-shaped curve.
SimCLR
SimCLR (Simple Framework for Contrastive Learning of Visual Representations) is a self-supervised learning method for computer vision in which a network is trained to recognise that two differently augmented…
SmoothGrad
SmoothGrad is a saliency map technique that reduces visual noise in gradient-based explanations of neural network predictions by averaging gradients over many noisy copies of the input.
Soft Actor-Critic
Soft Actor-Critic (SAC) is an off-policy, maximum-entropy deep reinforcement learning algorithm that trains a stochastic actor-critic to maximize expected reward plus the entropy of its own policy, so the…
Soft MoE
Soft MoE (Soft Mixture of Experts) is a fully differentiable variant of the sparse mixture of experts (MoE) layer.
Softmax
Softmax is a function that converts a finite vector of real-valued scores into a vector of positive numbers that sum to one.
Sophia (optimizer)
Sophia (a backronym for Second-order Clipped Stochastic Optimization with Adaptive estimator) is a stochastic second-order optimization algorithm introduced in May 2023 by Hong Liu, Zhiyuan Li, David Hall…
Sparse attention
Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence
Sparse autoencoder
A sparse autoencoder (SAE) is a neural network that adds a sparsity penalty to an autoencoder's training loss so that only a small number of hidden units activate for any given input, producing a wide
Sparsity
Sparsity is the property of a data structure, model, or representation in which most values are zero or near-zero
Speculative Decoding
Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…
Speech Recognition
Speech recognition, usually called automatic speech recognition (ASR), is the computational task of converting a spoken-language signal into a sequence of written symbols.
Staged training
Staged training (also called multi-stage training or phased training) is a strategy in machine learning where a model is trained in two or more distinct sequential phases, each with its own objective, data…
State space model (deep learning)
A state space model (SSM) in deep learning is a class of sequence model that maps an input sequence to an output sequence through a fixed-size latent state
Stochastic Gradient Descent (SGD)
Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples.
Stride
Stride is the step size by which a filter (or pooling window) moves across the input in a convolutional neural network (CNN): a stride of 1 shifts the filter one position at a time and visits every location…
Supervised fine-tuning
Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.
Swin Transformer
The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping windows and introduces a shifted window…
Tanh (hyperbolic tangent)
The hyperbolic tangent, written tanh, is a smooth, S-shaped activation function that maps any real number into the open interval $$(-1, 1)$$, passing through the origin so that tanh(0) = 0 .
Target Network
A target network is a separate, slowly updated copy of a neural network used in deep reinforcement learning to compute stable learning targets
Tensor
In machine learning, a tensor is a multi-dimensional array of numbers that serves as the fundamental data structure for representing and manipulating data.
Tensor Processing Unit (TPU)
A Tensor Processing Unit (TPU) is a family of custom application-specific integrated circuits developed by Google to accelerate machine-learning computation.
Tensor Shape
A tensor shape is a tuple of integers that describes the number of elements along each dimension (or axis) of a tensor.
TensorFlow
TensorFlow is an open-source software platform for numerical computation and machine learning.
Text summarization
Text summarization is the natural language processing (NLP) task of automatically producing a shorter version of one or more documents that preserves the most important information from the original text.