Pre-training
Pre-training is a stage of machine learning in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use.
Explore Training & Optimization through related topics and the articles other pages reference most.
Articles that also belong to these categories. Counts cover all of Training & Optimization.
Showing 61-80 of 80 articles
Pre-training is a stage of machine learning in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use.
Proximal Policy Optimization (PPO) is an on-policy policy gradient reinforcement learning algorithm that stabilizes training by clipping the policy update so the new policy stays close ("proximal") to the old…
Pruning is a family of techniques used in machine learning and artificial intelligence to remove parts of a model or search space that are estimated to be unnecessary for accuracy or optimality.
Regularization is any deliberate change to a learning problem or training algorithm intended to improve performance on data that were not used to fit the model.
"Regularization rate" is the terminology used in Google's Machine Learning Crash Course and Machine Learning Glossary
The reparameterization trick is a technique for computing low-variance gradients of an expectation with respect to the parameters of the distribution being sampled.
Representation learning is the area of machine learning concerned with getting a system to discover the features it needs for a task, instead of having people specify those features by hand.
Ridge regression is a method of estimating the coefficients of a linear regression model by adding a penalty proportional to the sum of squared coefficients (the squared L2 norm) to the ordinary least squares…
Ridge regularization, also known as L2 regularization or Tikhonov regularization, is a technique in statistics and machine learning that adds a squared L2-norm penalty to a model's loss function.
Shrinkage in machine learning and statistics is a regularization technique that deliberately pulls model coefficient estimates toward zero, or toward some other fixed shrinkage target
Simulated annealing is a probabilistic method for finding a good approximate solution to a global optimization problem, especially one with many local optima that trap ordinary hill-climbing methods.
Squared hinge loss (also called L2 hinge loss or L2-loss) is a loss function used in machine learning for classification tasks, most commonly in support vector machines (SVMs).
Squared loss, also called quadratic loss, L2 loss, or squared error loss, is a loss function that penalizes a prediction by the square of its error: for a true value $y$ and a prediction $\hat{y}$
Stochastic gradient descent (SGD) is a first-order optimization method that updates parameters using a gradient estimate computed from a randomly selected example or subset of examples.
Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained.
A termination condition, also called a stopping criterion, convergence criterion, or halting condition, is a rule that decides when an iterative algorithm should stop running.
Training compute is the total number of floating-point operations (FLOP) performed to train a machine learning model, normally counted over the forward and backward passes of the final training run.
A training run is a single, deliberate instance of training a neural network from scratch (or from a prior checkpoint) on a specified dataset, with a fixed compute budget, hardware allocation, and time horizon.
Wasserstein loss is a loss function for training generative models that measures the distance between two probability distributions as the Wasserstein-1 distance
Weight decay is a regularization technique used in training neural networks that shrinks every parameter toward zero by a small fraction on each update step