Hyperparameter

A hyperparameter is an input that configures a machine learning algorithm or modeling pipeline, rather than an output fitted by that algorithm in one training run. Learning rate, regularization strength, tree depth, kernel choice, batch size, and the number of retained components in a preprocessing step can all serve as hyperparameters. Their meaning is operational: a quantity is a hyperparameter when it is supplied to the learning procedure from outside the parameter-fitting step. The fitted weights, coefficients, split points, or other parameters returned by that procedure are model parameters.^[1]

Hyperparameters need not be chosen by hand, fixed forever, or independent of data. A search procedure can select them using validation data, a schedule can change them during training, and a higher-level optimization can differentiate through training to update them. What distinguishes them from ordinary model parameters is the level of the learning procedure at which they are controlled.^[2][3]

Definition and scope

A supervised learner can be represented as a map

$$\mathcal I:\mathcal D \times \Lambda \longrightarrow \mathcal H,\qquad (D,\lambda)\longmapsto \hat f_{D,\lambda},$$

where $D$ is a training set, $\Lambda$ is the hyperparameter configuration space, $\lambda$ is one configuration, and $\hat f_{D,\lambda}$ is the fitted model. In this view, the learner receives $\lambda$ as an input and returns the fitted model and its learned parameter vector as outputs.^[1] This input-output distinction is more reliable than common shortcuts such as "hyperparameters are numbers set before training."

Scroll sideways for more →

The boundary depends on how the procedure is organized. Consider a regularized model. If an inner optimizer fits weights while a validation loop selects the regularization coefficient, the weights are parameters and the coefficient is a hyperparameter. In an empirical-Bayes procedure, a related quantity may instead be estimated by maximizing marginal likelihood. In a differentiable outer loop, a regularization coefficient may be updated by a gradient even though it remains a hyperparameter relative to the inner weight-fitting problem.^[2][3] The terminology therefore describes a role in an inference or optimization hierarchy, not an immutable property of a symbol.

In Bayesian models, the same word is also used for quantities that parameterize priors or otherwise sit above model parameters. Such quantities can sometimes be integrated out or estimated by evidence maximization instead of being fixed by an external search.^[2] This use is compatible with the hierarchical idea, but machine-learning usage is broader: architecture choices, optimization controls, preprocessing decisions, and stopping rules can be hyperparameters even when no probabilistic prior is present.

Static, scheduled, and conditional hyperparameters

A static hyperparameter has one value for a complete training run. A fixed regularization coefficient is a common example. A scheduled hyperparameter is a rule or sequence whose value changes with training time, such as a learning rate schedule. The schedule's shape, breakpoints, or controller settings are themselves configuration choices. Population-based training goes further by changing selected hyperparameters in response to the performance of a population of concurrently trained models.^[10]

A conditional hyperparameter is active only under another choice. If a support vector machine uses a radial-basis-function kernel, its kernel width is relevant; for a linear kernel, that same field is inactive. If network depth is configurable, settings for layer $i$ are active only when the chosen depth includes that layer. Such dependencies produce hierarchical spaces that can be represented as trees or directed acyclic graphs rather than simple rectangular grids.^[1][2]

These distinctions matter because an optimizer that assumes every variable is always active can waste trials or fit a misleading surrogate. They also prevent a conceptual error: a hyperparameter configuration is not necessarily a short vector of independent real numbers.

Main classes of hyperparameters

Hyperparameters can be grouped by the part of the learning system they control. The categories overlap, and a single choice can affect prediction quality, training dynamics, and resource use at the same time.

Model family and capacity

Structural choices determine which functions the fitted model can represent. Examples include:

• the maximum depth or minimum leaf size of a decision tree;
• the number of neighbors and distance rule in k-nearest neighbors;
• the kernel family and associated kernel settings in a support vector machine;
• the number of layers, hidden width, attention heads, or connectivity pattern in a neural network;
• the number of components retained by principal component analysis when PCA is part of the predictive pipeline.

These choices influence capacity, but "larger" is not synonymous with "better." A more expressive model can reduce approximation error while increasing compute, estimation variance, or sensitivity to regularization. The useful setting depends on the data distribution, training procedure, metric, and available budget.

Optimization and training

Optimization hyperparameters control how learned parameters are fitted. They include the optimizer family, learning rate, batch size, momentum coefficients, gradient-clipping thresholds, initialization scales, training duration, and schedule settings. Some are meaningful only for particular optimizers. Others interact strongly: a learning rate that works for one batch size or normalization scheme need not work after either changes.

The training budget can also be treated as a hyperparameter, but it has two roles that should not be confused. It may be part of the final procedure, such as the maximum number of updates, or it may be a temporary fidelity used to compare configurations cheaply. A model selected after ten epochs is not automatically the configuration that would rank best after full training.

Regularization and objective design

Regularization hyperparameters control penalties or stochastic constraints intended to shape the fitted solution. Examples include a penalty coefficient, a dropout probability, label-smoothing strength, augmentation magnitude, or an early stopping rule. Loss weights and margins can also be hyperparameters when the outer procedure chooses them rather than fitting them as ordinary model parameters.

The distinction between mechanisms matters. Loshchilov and Hutter showed that adding an $L_2$ penalty to the loss and applying multiplicative weight decay are equivalent for ordinary stochastic gradient descent after an appropriate rescaling, but not generally for adaptive methods such as Adam. Their AdamW formulation decouples weight decay from the loss-gradient update.^[19] Consequently, a numerical "weight decay" value cannot be interpreted without recording the optimizer and implementation.

Data and pipeline choices

A predictive system usually contains more than a final estimator. Choices about missing-value handling, normalization, feature selection, resampling, augmentation, tokenization, and class balancing can be hyperparameters of the complete pipeline. The union of settings across data preprocessing, feature engineering, and modeling stages forms one configuration space.^[1]

Pipeline choices must be evaluated inside the validation procedure. For example, a feature selector should be fitted only on the training portion of each split. Fitting it once on all available data and then cross-validating the downstream estimator allows validation information to affect the pipeline before evaluation. That produces an optimistic estimate even if the final estimator never receives validation labels directly.^[1]

Resource, deployment, and decision constraints

Some tasks select configurations against more than predictive loss. Training time, inference latency, memory use, energy use, model size, or a fairness constraint can be an objective or a hard bound. The resulting task may be constrained or multi-objective rather than a search for one scalar optimum. A Pareto set records configurations for which no other observed configuration is at least as good on every objective and strictly better on one.^[2]

Whether a systems option should be called a hyperparameter depends on the target procedure. The number of data-loader workers normally changes throughput without changing the mathematical model, so it is an implementation control if prediction quality is the only objective. It becomes part of the configuration problem when resource use or nondeterministic execution is itself under study.

Examples across model families

The following table gives representative roles without prescribing ranges. Numerical ranges copied from another task, dataset, or software version can be poor search spaces for a new experiment.

Scroll sideways for more →

Software names sometimes obscure the distinction. An estimator API may label every constructor argument a "parameter," even though some are hyperparameters in the learning-theory sense. Conversely, a configuration field can merely control logging or hardware use. The scientific question is what procedure the field configures and whether its selection can affect the reported result.

Hyperparameters as a model-selection problem

For many supervised tasks, hyperparameter selection can be written as a two-level problem. Let $\theta$ denote ordinary model parameters and $\lambda$ the hyperparameters. A simplified inner problem is

$$\theta^*(\lambda)\in\operatorname*{arg\,min}_{\theta} \mathcal L_{\mathrm{train}}(\theta,\lambda).$$

The outer problem then selects

$$\lambda^*\in\operatorname*{arg\,min}_{\lambda\in\Lambda} \mathcal L_{\mathrm{valid}}\bigl(\theta^*(\lambda),\lambda\bigr).$$

This is a bilevel formulation because changing $\lambda$ changes the solution of the inner training problem. It also makes the data boundary explicit: training loss fits model parameters, while a separate validation objective compares configurations.^[3]

The equations are an abstraction, not a promise that either optimum is unique or exactly computed. Neural-network training is stochastic and usually terminates before a global optimum. Some learners have multiple fitted solutions, validation estimates are noisy, and categorical or conditional choices may make the outer space discontinuous. In practice, a configuration evaluation means running a specified training procedure under a specified data split, seed policy, budget, and metric.

The outer objective is part of the definition of the selected hyperparameter. Optimizing accuracy can select a different configuration from optimizing log loss, calibration, latency, or a weighted cost. Changing the validation split can also change the winner. A "best hyperparameter" therefore has no task-independent meaning.

Selection is data-dependent

Although a hyperparameter is an input to the inner learner, its selected value can be learned from data at the outer level. Random search, cross-validation, Bayesian optimization, and gradient-based HPO all use observations from configuration evaluations. Probst, Boulesteix, and Bischl describe tuning as a data-dependent, second-level optimization and distinguish it from fitting first-level model parameters.^[15]

This resolves a common apparent contradiction. The statement "hyperparameters are not learned from data" is true only if "learned" refers narrowly to the inner fitting algorithm. It is false for an end-to-end workflow in which validation results determine the configuration.

Final fitting and evaluation

After choosing $\lambda^*$, a practitioner may refit the model on the union of the training and validation data. That refit produces a new set of model parameters under the already selected configuration. A genuinely held-out test set can then estimate the performance of the complete selection-and-fitting procedure. If the test result influences another configuration decision, the test set has become part of the selection loop and is no longer an untouched final evaluation set.^[12][13]

Designing a configuration space

The search space is a substantive modeling decision. It specifies which configurations can be found and how a search budget is distributed among them.

Domains and scales

A hyperparameter domain can be continuous, integer-valued, binary, ordinal, or categorical. Bounds are usually imposed for practical reasons. A sampling distribution or transformation adds another layer: drawing a learning rate uniformly in its raw value is different from drawing its logarithm uniformly.

Logarithmic sampling is appropriate when plausible values span orders of magnitude and ratios are more informative than absolute differences. It is not a universal rule for every positive variable. A probability confined to $[0,1]$, a small tree depth, and a categorical optimizer choice require different representations. The bounds and sampling measure together encode where the search spends effort.^[1]

An excessively narrow space can exclude useful configurations. An excessively broad space can spend most trials in unstable, invalid, or prohibitively expensive regions. When multiple dimensions have broad ranges, the fraction of viable configurations can shrink rapidly. Search-space design should therefore use algorithm constraints, prior experiments, and resource limits, while leaving enough room to test whether a supposed default transfers.^[1]

Conditional and forbidden regions

Conditional structure should be explicit. A pipeline that chooses between a linear model and a tree model should activate only the settings belonging to the selected branch. Forbidden combinations can encode hard incompatibilities, such as a solver that does not support a selected penalty. Soft constraints can instead assign a cost or failure outcome, but a high rate of failed trials makes the search inefficient and complicates comparisons.

Conditionality also changes interpretation. Importance calculated across the entire space may make a branch-specific setting appear unimportant because it is inactive in most configurations. Analysis should state the domain and conditioning under which importance is measured.

Configuration versus fidelity

A fidelity is a resource level used as a proxy for a full evaluation. Common fidelities include training steps, data subset size, image resolution, or the number of cross-validation folds. Multi-fidelity methods allocate small budgets broadly and larger budgets to selected configurations.^[2][7]

Fidelity must predict enough about full-budget performance to be useful. Rankings can change during training, and a configuration that learns slowly can ultimately outperform an early leader. Early-stopping algorithms therefore make an assumption about the information contained in partial learning curves. That assumption should be checked for the workload rather than treated as a free speedup.

Search spaces are part of the reported method

Two studies can use the same search algorithm and budget yet reach different conclusions because their spaces differ. Reproducible reporting includes:

• every hyperparameter considered and which were fixed;
• the type, bounds, transformation, and sampling distribution for each domain;
• conditional and forbidden relationships;
• the objective, data splits, and aggregation rule;
• the number of trials, stopping rule, and per-trial resources;
• failure handling, seeds, and parallel execution policy.

Publishing only the winning configuration does not reveal the selection procedure or whether competing methods received comparable opportunities.

How hyperparameters affect learning

Hyperparameters influence several aspects of a learning system at once. Their effects are usually conditional rather than universal.

Optimization behavior

A learning rate that is too large for a particular optimizer and loss landscape can make updates unstable; one that is too small can require more steps than the available budget. Batch size affects gradient estimation, memory use, the number of updates per pass through the data, and attainable hardware parallelism. Training duration determines how long optimization and regularization mechanisms act.

These variables cannot always be studied one at a time. Goyal and colleagues used a linear learning-rate scaling rule plus gradual warmup when increasing the minibatch in a specific ResNet-50 ImageNet training system, reaching a batch size of 8,192 without losing the small-batch accuracy reported in their experiment.^[17] That result established a useful recipe in that setting, not a universal identity between batch size and learning rate.

A much larger experimental study by Shallue and colleagues measured 168,160 trained models across 35 workloads. It found substantial variation across workloads and argued that disagreements about large-batch training could often be traced to different hyperparameter-tuning protocols and compute budgets. The authors did not find evidence in their tested workloads that larger batches inherently reduced out-of-sample performance.^[18] Together, these studies support joint retuning and workload-specific measurement, not a fixed scaling law for every optimizer and architecture.

Generalization and regularization

Penalty strengths, data augmentation, dropout, early stopping, and capacity controls can alter generalization. Their effects depend on the rest of the procedure. More dropout is not always more protection against overfitting; it can also prevent the model from fitting useful structure. A deeper tree can lower training error while changing variance, but the result depends on sample size and other tree constraints.

Validation itself creates another layer of overfitting. Trying more configurations increases the opportunity to select one that benefited from noise in the validation estimate. The resulting configuration can look better on the selection criterion than it performs on fresh data.^[12]

Capacity and representation

Architecture and representation choices determine the hypothesis space exposed to the inner optimizer. Increasing width, depth, feature count, or retained dimensions can improve approximation while raising resource use and changing optimization. Insufficient capacity can cause underfitting, but excess capacity does not mechanically imply poor test performance. Regularization, data volume, inductive bias, and optimization all affect the outcome.

Neural architecture search treats architectural choices as a structured search problem. DARTS, for example, relaxed a discrete operation space into continuous architecture variables and optimized them with gradients in a bilevel formulation.^[22] This illustrates why "architecture choices cannot be differentiated" is not a sound definition of hyperparameters.

Computational cost

Hyperparameters can change cost before they change accuracy. Larger models use more memory and computation. More trees can increase training and inference time. Longer sequences can alter attention cost. Cross-validation multiplies fits. A configuration that is infeasible under the deployment constraint is not useful even if its unconstrained validation score is best.

For this reason, the target can be a constrained objective or a tradeoff rather than prediction loss alone. Resource measurements should use the intended hardware and software environment when systems performance is part of the claim.

Interactions

An interaction occurs when the effect of one hyperparameter depends on another. Frequent examples include:

Scroll sideways for more →

Interactions make sequential one-at-a-time tuning risky. They do not imply that every variable must always be searched jointly. A staged procedure can be reasonable when supported by prior evidence, but it should revisit coupled choices and report what was held fixed.

Hyperparameter importance and tunability

Hyperparameter importance asks how much variation in an outcome is associated with a setting over a specified space. It is not an intrinsic ranking that transfers automatically across datasets or search spaces.

Hutter, Hoos, and Leyton-Brown introduced an efficient functional-ANOVA analysis using random-forest performance models. It decomposes predicted performance variation into main effects and interactions over a defined configuration space. In their studied high-dimensional problems, much of the variation was attributable to a small number of hyperparameters.^[14] The result motivates prioritization, but it does not identify the same universally important variables for every task.

Probst, Boulesteix, and Bischl separated tunability from local importance. Their benchmark covered six algorithm families and 38 OpenML datasets and quantified the improvement available from tuning relative to defaults. They found that tunability and important settings varied by algorithm, dataset, and performance measure.^[15] A setting can have a large effect across a broad space but little benefit near a strong default, or matter only in interaction with another variable.

The baseline article cited one useful model-specific case. Greff and colleagues ran 5,400 LSTM experiments across speech recognition, handwriting recognition, and polyphonic music modeling. Within their chosen spaces and training protocol, learning rate accounted for more than two thirds of the modeled performance variance, hidden size ranked next, and momentum accounted for less than one percent except for a small interaction on one task.^[23] This finding is evidence about those LSTM experiments, not proof that learning rate is always the most important hyperparameter for every neural network.

Methods for analysis

Several analyses answer different questions:

• Functional ANOVA estimates main and interaction contributions over a declared space, often using a surrogate fitted to completed trials.^[14]
• Ablation from a reference configuration changes selected settings back toward a baseline and measures the resulting path. The answer can depend on the path when interactions exist.
• One-at-a-time sensitivity is easy to visualize but describes only a slice through the space at the fixed values of all other settings.
• Partial-dependence or surrogate plots summarize a fitted response model and inherit its approximation error and the sampling pattern of the trial archive.
• Local perturbation analysis tests robustness near one chosen configuration rather than global importance.

An importance plot should therefore state the metric, domain, conditioning, surrogate, and uncertainty. It should not be presented as a causal explanation of why the underlying learning algorithm works.

Hyperparameter tuning methods

The dedicated Hyperparameter Tuning article covers search algorithms in depth. This section provides the context needed to understand how a hyperparameter is selected.

Defaults and manual search

A software default is a configuration chosen for convenience or performance across some development distribution. It can be a useful baseline, but it is not guaranteed to be optimal for a new dataset, metric, or software version. A paper-derived configuration similarly carries assumptions about data, architecture, preprocessing, and budget.

Manual search uses human judgment to propose configurations. It can exploit domain knowledge and diagnose failures, but undocumented trial-and-error is hard to reproduce and can hide the effective number of comparisons. Recording every attempted configuration is necessary if a manual process contributes to the reported result.^[1]

Grid and random search

Grid search evaluates the Cartesian product of user-specified value sets. It is transparent and can be adequate for one or two small discrete domains, but its size grows multiplicatively. It also repeats the same coordinate values across many trials.

Random search samples configurations from declared distributions. Bergstra and Bengio compared it with grid and manual search on neural-network and deep-belief-network tasks. In their experiments, random search matched or improved on grid-configured neural networks with a fraction of the computation, and their analysis showed that different dimensions mattered on different datasets. They proposed random search as a reproducible baseline for adaptive methods.^[4]

The result is conditional on the search distributions and task. Random search does not know that a region is promising, and a poor domain remains poor however samples are drawn. Its practical strengths are simple parallel execution, flexible extension of the trial budget, and a clearly defined sampling process.^[2][4]

Model-based and Bayesian optimization

Bayesian optimization builds a surrogate from observed configurations and outcomes, then uses an acquisition rule to choose another evaluation. The surrogate can express predicted performance and uncertainty, allowing the acquisition rule to trade off sampling uncertain regions against refining promising ones.

Snoek, Larochelle, and Adams modeled validation performance with Gaussian processes and studied acquisition, variable evaluation cost, and parallel suggestions on several machine-learning tasks.^[6] Other surrogate families handle different spaces. The tree-structured Parzen estimator introduced by Bergstra and colleagues models promising and less-promising observations and was designed for conditional configuration spaces.^[5]

Bayesian optimization is not automatically superior. Performance depends on dimensionality, variable types, noise, conditional structure, parallelism, surrogate fit, and trial budget. The overhead of fitting and optimizing the surrogate should be included when wall-clock cost is compared.

Multi-fidelity and early-stopping methods

Successive-halving methods start multiple configurations at a small resource level, retain a fraction based on intermediate performance, and allocate more resource to survivors. Hyperband runs multiple allocation schedules to hedge between evaluating many configurations shallowly and fewer configurations deeply. Its original JMLR paper formulated this as adaptive resource allocation and reported more than an order-of-magnitude speedup over its competitor set on the studied deep-learning and kernel problems.^[7] That number is an experimental result for those benchmarks, not a general speed guarantee.

ASHA removes synchronization barriers from successive halving. Rather than waiting for every trial in a rung, it can promote eligible trials as results arrive. Li and colleagues designed it for large parallel settings and demonstrated a 500-worker experiment in their system study.^[8]

BOHB combines Hyperband's resource allocation with a model-based sampler inspired by tree-structured density estimation. Its ICML study reported stronger results than Hyperband and several Bayesian-optimization baselines across its benchmark collection.^[9] Such comparisons remain sensitive to implementation, search space, budget, and parallel scheduling.

Multi-fidelity evaluation can be inappropriate when low-budget results poorly predict the final ranking, when evaluation noise dominates early measurements, or when trials cannot be resumed consistently. The fidelity definition and promotion rule are part of the method.

Population-based training

Population-based training maintains several training runs. Periodically, poorly performing members can copy parameters and state from stronger members and perturb selected hyperparameters. Because these changes occur during training, the result is a learned schedule rather than one fixed setting. Jaderberg and colleagues evaluated the method in reinforcement learning, machine translation, and generative-model experiments.^[10]

PBT combines parameter training, model selection, and configuration adaptation, so its output cannot be summarized only by final scalar hyperparameters. Reproduction requires the population size, evaluation interval, exploit rule, perturbation distribution, and copied state.

Gradient-based and bilevel methods

If the outer objective changes smoothly with a continuous hyperparameter, a hypergradient can be obtained by differentiating through the training process or by implicit differentiation. Maclaurin, Duvenaud, and Adams reversed momentum-SGD dynamics to compute gradients of validation performance with respect to thousands of continuous hyperparameters, including schedules and regularization settings.^[11] Franceschi and colleagues placed such methods in a general bilevel framework.^[3]

These approaches can update many variables, but they require differentiability or a relaxation, careful treatment of memory and approximation, and a clear separation between training and validation objectives. They do not make discrete, conditional, or long-horizon selection problems disappear.

Architecture and pipeline search

When the configuration includes estimator choice, preprocessing branches, or architecture, hyperparameter tuning merges into AutoML and neural architecture search. The space becomes hierarchical because settings for an inactive branch have no effect.

This broader scope reinforces the operational definition: model family can be a categorical hyperparameter of a pipeline even though each family has its own learned parameters and lower-level hyperparameters. It also increases the risk of unfair comparisons if one family receives a wider, better-designed, or more computationally expensive search.

Validation and unbiased performance estimation

Hyperparameter selection and final performance estimation are different statistical tasks. Combining them on the same observations creates selection bias.

Training, validation, and test roles

A basic split assigns:

1. training data to fit model parameters for each configuration;
2. validation data to compare configurations;
3. test data to estimate the performance of the selected procedure.

When data are limited, cross-validation can replace a single validation split. Each candidate configuration is fitted on several training folds and scored on the corresponding held-out folds. The aggregation estimates its performance under that resampling protocol.

Cross-validation does not create independent data. Fold scores share observations and fitted training sets. The choice of split strategy must respect groups, time order, spatial dependence, or other sampling structure. Random folds are not valid merely because they are conventional.

Selection-induced optimism

Cawley and Talbot analyzed how the variance of a model-selection criterion permits overfitting at the selection level. Optimizing a noisy estimate can favor a configuration whose apparent advantage is partly noise, and using the selected criterion value as the final performance estimate is optimistic.^[12]

The size of the bias is not determined only by the number of trials. It also depends on criterion variance, correlation among candidates, search adaptivity, data size, and algorithm stability. A large search can be statistically safe with an independent final evaluation, while even a small search can be misleading if the same noisy score selects and certifies the model.

Nested resampling

Nested cross-validation separates selection from evaluation. Each outer split holds out an evaluation fold. Hyperparameters are selected using only the outer training portion, typically by an inner cross-validation. The selected configuration is refitted on that outer training portion and evaluated once on the outer test fold. Aggregating outer-fold results estimates the performance of the complete selection procedure.

Varma and Simon demonstrated the bias of using the minimum cross-validation error both to select settings and to report error in their simulation and microarray experiments. Their nested procedure substantially reduced that bias.^[13] Nested resampling does not identify one globally final configuration by itself; it evaluates the procedure that would select configurations on new training samples.

Pipeline containment

Every data-dependent operation that can influence predictions belongs inside the split. Imputation values, scaling statistics, selected features, oversampling, augmentation policies learned from data, and target transformations must be fitted or chosen using only the training portion of each relevant fold. Treating only the final estimator as "the model" is a common source of leakage.^[1]

Unsupervised preprocessing is not automatically safe. If it uses validation observations to estimate a representation or scale, it changes the information available to the downstream learner. Whether that creates material bias depends on the operation and sampling process, but the clean design is to contain it within the pipeline.

Stochastic variation

Neural-network initialization, data order, augmentation, split construction, and asynchronous execution can change observed results. Selecting the best seed is itself a selection operation. If a seed is searched for performance, it should be treated like another configuration variable rather than evidence that the method is reliably better.

Bouthillier and colleagues modeled variation from data sampling, initialization, and hyperparameter choice across five deep-learning tasks and architectures. They showed that these sources materially affected comparisons and recommended accounting for multiple sources of variation when evaluating improvements.^[16] Repeating only the winning configuration after a broad noisy search can understate uncertainty in the selection process.

Fair comparison of tuning procedures

A comparison should align the resource being limited:

• number of full configuration evaluations;
• total training steps or examples processed;
• accelerator time;
• wall-clock time at a stated parallelism;
• energy or financial cost.

These budgets are not interchangeable. A sequential model-based method may use fewer training runs but more coordination time. A massively parallel random search can finish sooner while consuming more total compute. Multi-fidelity methods evaluate many partial runs. Reporting the full resource ledger prevents "faster" from changing meaning between methods.

The same care applies to baselines. Each model family should receive a defensible search space, comparable information, and a stated budget. Tuning a proposed method extensively while using an untuned baseline confounds the learning algorithm with the quality of its selection process.

Defaults, transfer, and scaling

Prior experience can reduce the cost of choosing hyperparameters, but transfer is an empirical assumption.

Defaults and meta-learning

A default summarizes performance over some collection of tasks or developer experience. Probst and colleagues formalized data-based defaults as configurations that perform well across a benchmark distribution, while also showing that the gain from per-dataset tuning varies.^[15] A default is therefore a prior starting point, not a universal optimum.

Meta-learning can use results from earlier datasets to initialize a search on a new one. Auto-sklearn, for example, used dataset meta-features and prior benchmark results to warm-start its Bayesian optimizer with configurations that had performed well on similar datasets.^[20] This can save early trials when the similarity signal is useful. It can also transfer a poor bias when the new task, metric, preprocessing, or resource constraint differs.

Transfer across batch size or model scale

Some scaling recipes preserve useful configurations only after a deliberate reparameterization. The linear learning-rate rule in Goyal and colleagues was tested with a particular optimizer, architecture, data, batch range, warmup, and training schedule.^[17] Copying the rule without those conditions is not evidence-based transfer.

Maximal Update Parameterization, or muP, targets transfer across network scale. Yang and colleagues reported that several tuned hyperparameters remained stable across width under their parameterization. In their experiments, settings tuned on a 13-million-parameter proxy were transferred to a 350-million-parameter BERT-large setup, and settings from a 40-million-parameter proxy were transferred to a 6.7-billion-parameter GPT-3-style model. The paper reported competitive or better values than the published comparisons with substantially lower tuning cost.^[21]

Those are results under the paper's parameterization, architectures, objectives, and comparisons. They do not show that arbitrary hyperparameters transfer across every change in depth, data, token budget, optimizer, or model family. The same paper distinguishes transferable and non-transferable quantities, and implementation checks are required for the intended scaling rules.

A transfer checklist

Before reusing a configuration, compare:

• task, target distribution, and evaluation metric;
• preprocessing and representation;
• model family and parameterization;
• optimizer implementation and numerical precision;
• batch construction and total training budget;
• hardware constraints and parallelism;
• data volume and augmentation;
• software defaults and version.

When any of these change materially, a local validation sweep or robustness analysis is stronger evidence than an assertion that the old optimum transfers.

Reproducible practice

A defensible hyperparameter study records the selection process, not only the final values.

Before the search

Define the prediction task, data-access boundaries, primary metric, constraints, and resource budget. Freeze a test policy. Specify which parts of the pipeline are tunable and why each domain is plausible. Identify conditional branches and invalid configurations. Decide how failed or interrupted trials will count.

During the search

Log each proposed configuration, code and environment version, data split, seed, resource allocation, intermediate metric, final metric, termination reason, and elapsed or compute time. Preserve unsuccessful trials because deleting them changes the apparent search history. If the search space or objective changes, record a new phase rather than silently merging incompatible trials.

Parallel systems also need scheduler details. Asynchronous methods can observe results in a hardware-dependent order, and that order can affect later proposals. Exact determinism may be impractical, but the mechanism and observed variability can still be documented.

After selection

Report:

• the winning configuration and complete search space;
• the selection metric and its uncertainty;
• the total search budget and parallelism;
• robustness to nearby settings when feasible;
• results across independent seeds or splits appropriate to the claim;
• the final evaluation protocol;
• any refitting on combined training and validation data;
• all deviations from the preregistered or initial plan.

A single best validation score is not a confidence interval. A single final seed is not evidence that seed variation is negligible. If compute prevents extensive repetitions, state that limitation rather than replacing missing uncertainty with certainty.

Common misconceptions

"Hyperparameters are always chosen by a human." They can be set by defaults, random or model-based search, population controllers, meta-learning, or outer gradients.

"Hyperparameters must be fixed before training." Static settings are common, but schedules and population-based methods change settings during a run.^[10]

"Hyperparameters are not learned from data." They are not outputs of the inner fitting step, but selection commonly uses validation data and is a data-dependent outer procedure.^[15]

"A hyperparameter cannot be differentiable." Gradient-based HPO and differentiable architecture search use continuous hyperparameters or relaxations and differentiate an outer objective through training.^[3][11][22]

"The validation winner estimates its own test performance." Selection favors favorable validation noise. A separate test or nested outer loop is needed to estimate the selected procedure without reusing the selection criterion.^[12][13]

"Random search is always better than grid search." Bergstra and Bengio established advantages in their studied high-dimensional settings and motivated random search as a baseline. A small, meaningful discrete grid can still be appropriate, and both methods depend on the specified domain.^[4]

"Bayesian optimization is always the most sample-efficient choice." Its result depends on surrogate adequacy, noise, space structure, parallelism, budget, and overhead. No one optimizer dominates every HPO problem.^[1][2]

"Larger batches inherently generalize worse." Shallue and colleagues found no such inherent degradation in their tested workloads after considering tuning and budget, while also finding large workload-to-workload variation.^[18]

"There is one universal range for each hyperparameter." Useful values depend on model, data, parameterization, optimizer, metric, and budget. Published ranges are evidence only under their documented conditions.

"The random seed is a harmless knob to optimize." Choosing a seed by outcome converts randomness into selection and can exaggerate performance. Seeds should normally sample variability, not serve as performance knobs.^[16]

"The hyperparameter with the largest global importance must be tuned first." Importance depends on the declared space and can differ from the improvement available near a strong default. Interactions and local robustness can matter more for the actual decision.^[14][15]

See also

• Hyperparameter Tuning
• Bayesian Optimization
• Cross-Validation
• AutoML
• Neural architecture search
• Learning Rate
• Regularization
• Bias-variance tradeoff

References

1. ^Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A.-L., Deng, D., and Lindauer, M. (2023). "Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges." *WIREs Data Mining and Knowledge Discovery*, 13(2), e1484. doi.org/...widm.1484
2. ^Feurer, M., and Hutter, F. (2019). "Hyperparameter Optimization." In *Automated Machine Learning: Methods, Systems, Challenges*, pp. 3-38. Springer. doi.org/...978-3-030-05318-5_1
3. ^Franceschi, L., Frasconi, P., Salzo, S., Grazzi, R., and Pontil, M. (2018). "Bilevel Programming for Hyperparameter Optimization and Meta-Learning." *Proceedings of Machine Learning Research*, 80, 1568-1577. proceedings.mlr.press/...franceschi18a
4. ^Bergstra, J., and Bengio, Y. (2012). "Random Search for Hyper-Parameter Optimization." *Journal of Machine Learning Research*, 13, 281-305. jmlr.org/...bergstra12a
5. ^Bergstra, J., Bardenet, R., Bengio, Y., and Kegl, B. (2011). "Algorithms for Hyper-Parameter Optimization." *Advances in Neural Information Processing Systems*, 24, 2546-2554. papers.nips.cc/...for-hyper-parameter-optimization
6. ^Snoek, J., Larochelle, H., and Adams, R. P. (2012). "Practical Bayesian Optimization of Machine Learning Algorithms." *Advances in Neural Information Processing Systems*, 25. papers.nips.cc/...n-of-machine-learning-algorithms
7. ^Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2018). "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization." *Journal of Machine Learning Research*, 18(185), 1-52. jmlr.org/...16-558
8. ^Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Ben-tzur, J., Hardt, M., Recht, B., and Talwalkar, A. (2020). "A System for Massively Parallel Hyperparameter Tuning." *Proceedings of Machine Learning and Systems*, 2. proceedings.mlsys.org/...a6b171c71b88bbfc-Abstract
9. ^Falkner, S., Klein, A., and Hutter, F. (2018). "BOHB: Robust and Efficient Hyperparameter Optimization at Scale." *Proceedings of Machine Learning Research*, 80, 1437-1446. proceedings.mlr.press/...falkner18a
10. ^Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K. (2017). "Population Based Training of Neural Networks." arXiv:1711.09846. arxiv.org/...1711.09846
11. ^Maclaurin, D., Duvenaud, D., and Adams, R. P. (2015). "Gradient-based Hyperparameter Optimization through Reversible Learning." *Proceedings of Machine Learning Research*, 37, 2113-2122. proceedings.mlr.press/...maclaurin15
12. ^Cawley, G. C., and Talbot, N. L. C. (2010). "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation." *Journal of Machine Learning Research*, 11, 2079-2107. jmlr.org/...cawley10a
13. ^Varma, S., and Simon, R. (2006). "Bias in error estimation when using cross-validation for model selection." *BMC Bioinformatics*, 7, 91. doi.org/...1471-2105-7-91
14. ^Hutter, F., Hoos, H., and Leyton-Brown, K. (2014). "An Efficient Approach for Assessing Hyperparameter Importance." *Proceedings of Machine Learning Research*, 32(1), 754-762. proceedings.mlr.press/...hutter14
15. ^Probst, P., Boulesteix, A.-L., and Bischl, B. (2019). "Tunability: Importance of Hyperparameters of Machine Learning Algorithms." *Journal of Machine Learning Research*, 20(53), 1-32. jmlr.org/...18-444
16. ^Bouthillier, X., Delaunay, P., Bronzi, M., Trofimov, A., Nichyporuk, B., Szeto, J., Sepah, N., Raff, E., Madan, K., Voleti, V., Ebrahimi Kahou, S., Michalski, V., Serdyuk, D., Arbel, T., Pal, C., Varoquaux, G., and Vincent, P. (2021). "Accounting for Variance in Machine Learning Benchmarks." *Proceedings of Machine Learning and Systems*, 3. proceedings.mlsys.org/...89f858a1d9f5c1eb-Abstract
17. ^Goyal, P., Dollar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. (2017). "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour." arXiv:1706.02677. arxiv.org/...1706.02677
18. ^Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E. (2019). "Measuring the Effects of Data Parallelism on Neural Network Training." *Journal of Machine Learning Research*, 20(112), 1-49. jmlr.org/...18-789
19. ^Loshchilov, I., and Hutter, F. (2019). "Decoupled Weight Decay Regularization." *International Conference on Learning Representations*. arxiv.org/...1711.05101
20. ^Feurer, M., Klein, A., Eggensperger, K., Springenberg, J. T., Blum, M., and Hutter, F. (2015). "Efficient and Robust Automated Machine Learning." *Advances in Neural Information Processing Systems*, 28. proceedings.neurips.cc/...3f79975ec59a3a6-Abstract
21. ^Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. (2022). "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer." arXiv:2203.03466. arxiv.org/...2203.03466
22. ^Liu, H., Simonyan, K., and Yang, Y. (2019). "DARTS: Differentiable Architecture Search." *International Conference on Learning Representations*. arxiv.org/...1806.09055
23. ^Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., and Schmidhuber, J. (2017). "LSTM: A Search Space Odyssey." *IEEE Transactions on Neural Networks and Learning Systems*, 28(10), 2222-2232. doi.org/...TNNLS.2016.2582924