Machine Learning

Sparse Representation

Sparse representation is a data encoding strategy in which most values in a vector, matrix, or tensor are zero (or near zero), with only a small fraction of elements carrying non-zero values.

Data & Datasets

Sparse Vector

A sparse vector is a vector in which most of the elements are zero, in contrast to a dense vector, in which most elements are non-zero.

Computer ScienceMathematics

Sparse attention

Sparse attention is a family of techniques that cut the computational and memory cost of the attention mechanism in transformer models by letting each token attend to only a subset of other tokens in a sequence

Deep LearningModel Architecture

Sparse autoencoder

A sparse autoencoder (SAE) is a neural network that adds a sparsity penalty to an autoencoder's training loss so that only a small number of hidden units activate for any given input, producing a wide

Deep LearningInterpretability

Sparse upcycling

Sparse upcycling is a technique for building a sparsely activated mixture of experts (MoE) model by initializing it from an already trained dense Transformer checkpoint

Reinforcement Learning

Sparsity

Sparsity is the property of a data structure, model, or representation in which most values are zero or near-zero

Deep Learning

Spatial Pooling

Spatial pooling is a downsampling operation in convolutional neural networks (CNNs) that replaces a local region of a feature map with a single summary statistic, such as the maximum or the average of the…

Computer VisionNeural Networks

Speculative Decoding

Speculative decoding is a lossless inference acceleration technique for autoregressive transformer models in which a small, fast draft model proposes several future tokens at once and the larger target model…

AI InferenceDeep Learning

Speech Recognition

Speech recognition, usually called automatic speech recognition (ASR), is the computational task of converting a spoken-language signal into a sequence of written symbols.

Deep LearningNatural Language Processing

SpinQuant

SpinQuant is a post-training quantization method for large language models that inserts learned rotation matrices into a transformer network to make its weights, activations, and KV cache easier to represent…

AI Infrastructure

Split

A split in machine learning is a partitioning operation, and the word names two distinct things.

Splitter

A splitter is a term used in two distinct senses in machine learning. The first and most common sense is a utility that partitions a dataset into subsets such as training, validation, and test sets, or into…

Model Evaluation

Squared Hinge Loss

Squared hinge loss (also called L2 hinge loss or L2-loss) is a loss function used in machine learning for classification tasks, most commonly in support vector machines (SVMs).

Training & Optimization

Squared Loss

Squared loss, also called quadratic loss, L2 loss, or squared error loss, is a loss function that penalizes a prediction by the square of its error: for a true value $y$ and a prediction $\hat{y}$

StatisticsTraining & Optimization

Stability

Stability in machine learning is the property that a learning algorithm or trained model produces similar outputs, whether predictions, parameter values, or loss curves, when its inputs are perturbed slightly.

State (Reinforcement Learning)

In reinforcement learning (RL), a state is a complete description of the environment at a particular point in time, containing all the information an agent needs to choose its next action.

Reinforcement Learning

State space model (deep learning)

A state space model (SSM) in deep learning is a class of sequence model that maps an input sequence to an output sequence through a fixed-size latent state

Deep Learning

State-Action Value Function

The state-action value function, written Q^π(s, a) and also called the action-value function or Q-function, gives the expected discounted return an agent obtains by taking action a in state s and thereafter…

Reinforcement Learning

Static

In machine learning, static means offline (also called batch): the model is trained once on a fixed dataset, or its predictions are computed in advance and cached, rather than being updated or generated…

Stationarity

Stationarity is a property of a time series or stochastic process whose statistical characteristics, such as the mean, variance, and autocovariance, do not change when the series is shifted in time.

Statistics

Statistical learning theory

Statistical learning theory (SLT) is the mathematical framework that explains when and why machine learning algorithms generalize from a finite training sample to unseen data.

MathematicsStatistics

Stefano Ermon

Stefano Ermon is an Italian computer scientist and an associate professor of computer science at Stanford University, best known for foundational work on score-based generative models

Generative AIPeople

Stochastic Gradient Descent (SGD)

Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples.

Deep LearningTraining & Optimization

Stride

Stride is the step size by which a filter (or pooling window) moves across the input in a convolutional neural network (CNN): a stride of 1 shifts the filter one position at a time and visits every location…

Computer VisionDeep Learning

Structural risk minimization (SRM)

Structural risk minimization (SRM) is an inductive principle in statistical learning theory for selecting a learned model that simultaneously fits the training data and controls the capacity of the hypothesis…

Subsampling

Subsampling is the practice of drawing a smaller subset from a larger collection of data points, training examples, features, or signal values, in order to cut compute cost, add regularization, or rebalance a…

Statistics

Summary

In TensorFlow, a summary is a piece of data written to disk during training so that it can later be visualized in TensorBoard. The tf.summary module is the API used to record those values.

Supervised Learning

Supervised learning is a machine learning paradigm in which a model is fitted to examples that pair an input with an observed target.

Artificial Intelligence

Supervised Machine Learning

Supervised machine learning is a branch of machine learning in which a model learns a mapping from inputs to outputs by training on labeled examples, where every input (a feature vector) is paired with the…

Supervised fine-tuning

Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.

Artificial IntelligenceDeep Learning

Support Vector Machine (SVM)

A support vector machine (SVM) is a family of supervised learning methods used for classification and regression.

Synthetic Feature

A synthetic feature (also called a constructed feature or derived feature) is a variable created by transforming, combining, or otherwise manipulating one or more existing features in a dataset, rather than…

Data & DatasetsData Science

TIES-Merging

TIES-Merging is a training-free model merging method that combines several models fine-tuned from a shared pre-trained checkpoint into one multitask model while explicitly resolving the interference between…

Reinforcement Learning

TPU Board

A TPU board (Tensor Processing Unit board) is a printed circuit board (PCB) that houses one or more Tensor Processing Unit chips along with associated memory, power delivery, and interconnect components.

AI HardwareGoogle

TPU Chip

This article covers the TPU chip and its hardware. For the broader topic, see Tensor Processing Unit (TPU).

AI HardwareGoogle

TPU Device

A TPU device is a Google-designed application-specific integrated circuit (ASIC), the physical Tensor Processing Unit chip and its host hardware

AI HardwareGoogle

TPU Master

The TPU master is the control and coordination role in Google's Tensor Processing Unit (TPU) system that drives the actual TPU computation.

AI HardwareGoogle

TPU Node

A TPU node is the legacy Google Cloud architecture for accessing Tensor Processing Unit (TPU) hardware, in which a user's virtual machine (VM) runs application code and communicates with a separate

AI HardwareAI Infrastructure

TPU Pod

A TPU Pod is a single Google supercomputer built from many Tensor Processing Unit (TPU) chips wired directly to each other by a high-speed Inter-Chip Interconnect (ICI) fabric arranged as a 2D or 3D torus, so…

AI ComputeAI Hardware

TPU Slice

A TPU slice is a collection of Tensor Processing Unit (TPU) chips that all sit inside the same Google Cloud TPU Pod and are connected to one another by a high-speed inter-chip interconnect (ICI).

AI HardwareGoogle

TPU Type

The types of TPU (Tensor Processing Unit) are the successive generations of custom machine-learning accelerator that Google has built since 2015: v1, v2, v3, v4, v5e, v5p, v6e (Trillium), and v7 (Ironwood).

AI HardwareGoogle

TPU Worker

A TPU worker is a virtual machine (VM) running Linux that has direct access to one or more Tensor Processing Unit (TPU) chips and executes the actual TPU computation on that attached hardware.

AI HardwareAI Infrastructure

TPU resource

A TPU resource is an allocation of Tensor Processing Unit compute, Google's custom machine learning accelerator chips, that you reserve and run as a unit

Tabular Classification Models

Tabular classification models predict a discrete label, or a probability distribution over labels, from rows of a table.

AI Models

Tabular Q-Learning

Tabular Q-learning is the classic form of Q-learning, a model-free reinforcement learning algorithm that stores the action-value function Q(s, a) explicitly in a lookup table, the Q-table

Reinforcement Learning

Tabular Regression Models

Tabular regression models are machine learning systems that predict a continuous numeric target from a vector of tabular features, where rows are samples and columns are heterogeneous attributes (numeric…

AI Models

Tabular models

Tabular models are machine learning systems that learn from data arranged in tables, where each row is a sample and each column is a feature.

Data Science

Target

In supervised learning, the target is the variable that a model learns to predict from input features.

Target Network

A target network is a separate, slowly updated copy of a neural network used in deep reinforcement learning to compute stable learning targets

Deep LearningReinforcement Learning

Task arithmetic

Task arithmetic is a model-editing technique that steers the behavior of a neural network by adding or subtracting vectors in its weight space.

Reinforcement Learning

Temperature (artificial intelligence)

Temperature is a hyperparameter that controls the randomness of a large language model's output by scaling the model's raw scores, called logits.

Large Language ModelsNatural Language Processing

Temporal data

Temporal data is data where each observation is tagged with a timestamp, so the order in which observations arrive carries meaning.

Statistics

Temporal-difference learning

Temporal-difference (TD) learning is a class of model-free reinforcement learning methods that learn value-function estimates by bootstrapping: updating each estimate of how good a state is toward a target…

AlgorithmsReinforcement Learning

Tensor

In machine learning, a tensor is a multi-dimensional array of numbers that serves as the fundamental data structure for representing and manipulating data.

Deep LearningMathematics

Tensor Rank

The rank of a tensor, also referred to as its order or degree, is the number of dimensions (axes or indices) needed to describe the tensor.

Mathematics

Tensor Shape

A tensor shape is a tuple of integers that describes the number of elements along each dimension (or axis) of a tensor.

Deep LearningMathematics

Tensor size

The size of a tensor is a description of how big the tensor is, and the term carries two distinct meanings in everyday deep learning.

TensorBoard

TensorBoard is the open-source visualization toolkit for TensorFlow, described by Google as "a suite of visualization tools to understand, debug, and optimize TensorFlow programs for ML experimentation." It…

Data ScienceDeveloper Tools

TensorFlow

TensorFlow is an open-source software platform for numerical computation and machine learning.

AI Tools & ProductsDeep Learning

TensorFlow Decision Forests (TF-DF)

TensorFlow Decision Forests (often abbreviated TF-DF) is an open-source Google library for training, serving, and interpreting decision-forest models such as Random Forest and Gradient Boosted Decision Trees…

Developer ToolsGoogle