Scalar

See also: vector, matrix, tensor, linear algebra, gradient descent

A scalar is a single numerical value, a quantity with magnitude but no direction, and the simplest object in linear algebra and machine learning. In the standard reference text Deep Learning, Goodfellow, Bengio, and Courville define it directly: "A scalar is just a single number, in contrast to most of the other objects studied in linear algebra, which are usually arrays of multiple numbers." ^[1] Equivalently, a scalar is a rank-0 tensor: it has zero dimensions and is represented by exactly one number, with shape () in NumPy and torch.Size([]) in PyTorch. ^[13][15] This contrasts with a vector (rank 1), a matrix (rank 2), and higher-rank tensors.

What is a scalar?

A scalar is a single numerical value that represents a quantity with magnitude but no direction. In mathematics and linear algebra, a scalar is formally defined as an element of a field that is used to define a vector space through scalar multiplication. ^[1][12] In machine learning and deep learning, scalars are the most basic data type, serving as the building blocks from which vectors, matrices, and tensors are constructed. ^[1]

Scalars occupy the lowest rank in the hierarchy of mathematical objects used in computation. A scalar is a rank-0 tensor, meaning it has zero dimensions and requires only a single number to represent it, regardless of the dimensionality of the surrounding space. ^[1][7] While scalars may seem simple compared to vectors or matrices, they appear throughout virtually every machine learning algorithm: as hyperparameters like the learning rate, as the output of loss functions, as individual weights and biases in neural networks, and as evaluation metrics such as accuracy or precision.

The word "scalar" derives from the Latin scalaris, meaning "of or pertaining to a ladder," which itself comes from scala ("a flight of steps, ladder, scale"). The French mathematician Francois Viete first recorded the mathematical usage in 1591. The Irish mathematician William Rowan Hamilton introduced the term into English in 1846, using it to describe the real part of a quaternion. ^[3] Hamilton wrote that the real part of a quaternion "may receive all values contained on the one scale of progression of numbers from negative to positive infinity," and so he called it the "scalar part." ^[3] The name reflects the idea that a scalar sits on a single number line, or scale, in contrast to quantities that carry directional information.

ELI5: Explain like I'm 5

Imagine you have a box of crayons. If someone asks "how many crayons do you have?" and you answer "eight," that number is a scalar. It is just one number that tells you how much of something there is.

Now imagine you are pointing at a tree and saying "the tree is 20 steps away in that direction." That is not a scalar because it has both a number (20 steps) and a direction (where you are pointing). Things with both a number and a direction are called vectors.

In machine learning, computers use lots of scalars to learn things. Each scalar is like one tiny knob the computer can turn up or down to get better at a task, like recognizing a picture of a cat or translating a sentence from English to French.

How is a scalar defined in linear algebra?

Formal definition in linear algebra

In linear algebra, a scalar is an element of the underlying field F over which a vector space V is defined. ^[12] A vector space is a set of vectors equipped with two operations: vector addition and scalar multiplication. Scalar multiplication takes a scalar a from the field F and a vector v from V and produces another vector av in V.

The field F can be any of several standard number systems:

Scroll sideways for more →

A field must satisfy the standard axioms of addition and multiplication: commutativity, associativity, distributivity, and the existence of identity and inverse elements. ^[12] When the algebraic structure is relaxed from a field to a ring (which may lack multiplicative inverses), the resulting structure is called a module rather than a vector space, and the "scalars" are elements of that ring.

Scalars as rank-0 tensors

In the language of tensor algebra, mathematical objects are classified by their rank (also called order or degree): ^[7]

Scroll sideways for more →

A scalar is a rank-0 tensor. It is invariant under coordinate transformations, meaning that the numerical value of a scalar does not change when the coordinate system is rotated or translated. ^[7] This property distinguishes scalars from vectors and higher-rank tensors, whose components change under such transformations even though the underlying geometric or physical quantity remains the same.

How are scalars written? Notation conventions

Standard mathematical notation uses specific typographical conventions to distinguish scalars from other mathematical objects:

Scroll sideways for more →

In machine learning literature, the convention from Goodfellow, Bengio, and Courville's Deep Learning textbook is widely followed. As the authors state, "We write scalars in italics. We usually give scalars lowercase variable names." ^[1] Set membership is denoted with notation such as s in R (meaning s is a real-valued scalar) or n in N (meaning n is a natural number), as in the book's own examples "Let s in R be the slope of the line" and "Let n in N be the number of units." ^[1]

What operations can you do with scalars?

Arithmetic operations

Scalars obey the standard arithmetic operations inherited from their underlying field: ^[12]

Scroll sideways for more →

These operations are commutative (for addition and multiplication), associative, and satisfy the distributive law. Division is defined for all nonzero scalars in a field.

What is scalar multiplication?

Scalar multiplication is one of the two fundamental operations that define a vector space. ^[12] When a scalar c multiplies a vector v = [v_1, v_2, ..., v_n], the result is a new vector whose every component is scaled by c:

$$c \mathbf{v} = [c v_1, c v_2, \ldots, c v_n]$$

Geometrically, this operation stretches or contracts the vector by a factor of |c|. If c is positive, the resulting vector points in the same direction as v. If c is negative, the direction reverses. If c = 0, the result is the zero vector. ^[2]

Scalar multiplication of a matrix works the same way: each element of the matrix is multiplied by the scalar. For a scalar $c$ and a matrix $\mathbf{A}$ with entries $a_{ij}$, the product $c\mathbf{A}$ has entries $c \, a_{ij}$. This operation is commutative, meaning $c\mathbf{A} = \mathbf{A}c$.

Scalar multiplication is distributive over both vector and matrix addition: ^[12]

• c(u + v) = cu + cv
• (a + b)v = av + bv

Scalar product (dot product)

The scalar product, also known as the dot product or inner product, is an operation that takes two vectors of equal length and returns a scalar. ^[2] For two vectors a = [a_1, a_2, ..., a_n] and b = [b_1, b_2, ..., b_n], the dot product is defined algebraically as:

$$\mathbf{a} \cdot \mathbf{b} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n$$

Geometrically, the dot product equals the product of the two vectors' magnitudes and the cosine of the angle between them:

$$\mathbf{a} \cdot \mathbf{b} = |\mathbf{a}| \, |\mathbf{b}| \cos(\theta)$$

The dot product has several properties: it is commutative (a . b = b . a), distributive over vector addition, and compatible with scalar multiplication. The result is always a scalar, which is why this operation is called the "scalar product." The dot product appears throughout machine learning, from computing neuron activations to measuring similarity between embeddings.

Where do scalars appear in machine learning?

Model parameters: weights and biases

In neural networks, every connection between neurons is associated with a scalar weight, and every neuron typically has a scalar bias term. For a single neuron receiving n inputs, the output before the activation function is computed as:

$$z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b$$

Here, each $w_i$ is a scalar weight, each $x_i$ is a scalar input feature, and $b$ is a scalar bias. The weighted sum $z$ is also a scalar. After applying a nonlinear activation function (such as ReLU, sigmoid, or tanh), the output is again a scalar that gets passed to the next layer.

During training, these scalar weights and biases are adjusted iteratively by optimization algorithms like gradient descent, Adam, or SGD to minimize the loss function. ^[14] A modern large language model may have billions of individual scalar parameters.

Which hyperparameters are scalars?

Many of the settings that control how a model trains are scalar values. These are called hyperparameters because they are not learned from data but are set by the practitioner before training begins.

Scroll sideways for more →

The learning rate is widely regarded as the single most important hyperparameter to tune. If the learning rate is too large, gradient descent may overshoot minima and diverge. If it is too small, training converges slowly and may get stuck in poor local minima. ^[14]

Why must a loss function return a scalar?

A loss function (also called a cost function or objective function) maps the predictions of a model and the ground-truth labels to a single scalar value that measures how poorly the model is performing. Training a machine learning model is fundamentally the process of minimizing this scalar loss. ^[1]

Common loss functions include:

Scroll sideways for more →

The fact that the loss must be a scalar is not arbitrary. Automatic differentiation frameworks (such as PyTorch autograd and TensorFlow GradientTape) compute gradients by starting from a scalar output and propagating backward through the computational graph via the chain rule. ^[13] If the loss were a vector or matrix, the system would need to compute a full Jacobian rather than a single gradient vector, which is far more expensive. Reducing the loss to a scalar is what makes backpropagation efficient. ^[5]

Evaluation metrics

Model performance is typically summarized using scalar evaluation metrics:

Scroll sideways for more →

Each of these metrics distills the model's behavior over an entire dataset into a single scalar value, making it easy to compare models and track performance across experiments.

Gradient descent and scalar calculus

The gradient of a scalar-valued function is a vector that points in the direction of the steepest increase of that function. In gradient descent, the model parameters are updated in the opposite direction of the gradient to reduce the loss:

$$\theta_{\text{new}} = \theta_{\text{old}} - \alpha \nabla L(\theta_{\text{old}})$$

Here, $\alpha$ is the scalar learning rate, $L$ is the scalar-valued loss function, and $\nabla L$ is the gradient vector. The update rule multiplies the gradient vector by the scalar learning rate, demonstrating a direct application of scalar-vector multiplication. ^[14]

In backpropagation, the chain rule is used to compute the gradient of the scalar loss with respect to every parameter in the network. ^[5] Because the loss is a scalar, each partial derivative is also a scalar, and these partial derivatives are assembled into the gradient vector. This scalar-to-scalar differentiation at each step of the chain rule is what makes backpropagation computationally tractable.

What is a scalar field?

A scalar field is a function that assigns a scalar value to every point in a given region of space. In physics, common examples include temperature distributions, pressure fields, gravitational potential, and electric potential. At any given point in the field, the value is a scalar (a single number with no direction).

In machine learning, the loss function can be understood as a scalar field defined over the parameter space. Each point in parameter space corresponds to a particular set of model weights, and the loss function assigns a scalar value (the loss) to that point. The goal of training is to find the point in this scalar field where the value is minimized. The gradient of the loss function at any point in parameter space is a vector that points uphill, and gradient descent moves in the opposite direction.

The concept of a scalar field also appears in feature engineering and data visualization. A heatmap, for instance, is a visual representation of a scalar field over a two-dimensional domain, where color intensity represents the scalar value at each point.

How are scalars represented in NumPy, PyTorch, and TensorFlow?

In deep learning frameworks, scalars are represented as zero-dimensional tensors. A tensor that contains only one number is called a scalar, scalar tensor, rank-0 tensor, or 0D tensor; in NumPy it has ndim == 0 and shape (). ^[15] The table below shows how scalars are created and manipulated in several popular libraries.

Scroll sideways for more →

The TensorFlow documentation defines the simplest tensor concisely: "A scalar contains a single value, and no 'axes'." ^[16] In PyTorch, the .item() method extracts a standard Python number from a zero-dimensional tensor; this method requires the tensor to be a scalar. ^[13] It is commonly used when logging scalar metrics like the loss value during training. In TensorFlow, a rank-0 tensor behaves like a scalar (for example, tf.constant(4) prints with shape=()) and can be converted to a Python float via .numpy(). ^[16]

When a scalar is combined with a tensor in an arithmetic operation, most frameworks apply broadcasting: the scalar is logically expanded to match the shape of the tensor, and the operation is performed element-wise. For example, multiplying a 3x3 matrix by the scalar 2 produces a new 3x3 matrix where every element is doubled.

What is scalar quantization?

Scalar quantization is a technique used in model compression to reduce the size and computational cost of neural networks. ^[9] It works by mapping the continuous range of floating-point parameter values to a discrete set of fixed-point or integer values with fewer bits.

In a standard deep learning model, weights and activations are stored as 32-bit floating-point numbers (FP32). Scalar quantization reduces the precision of each individual scalar parameter, for example from 32 bits down to 16 bits (FP16), 8 bits (INT8), or even 4 bits (INT4). The mapping from a continuous scalar value to its quantized representation follows the formula: ^[9]

$$q = \operatorname{round}\left(\frac{x - \text{zero\_point}}{\text{scale}}\right)$$

where x is the original scalar value, scale and zero_point are scalar calibration parameters, and q is the resulting quantized integer.

Scroll sideways for more →

Scalar quantization is widely used for deploying large language models on consumer hardware. Formats like GGUF and GPTQ use various quantization schemes to shrink models that would otherwise require high-end GPUs.

What is mixed precision training?

Mixed precision training uses scalars of different numerical precisions within the same training run. Most computations are performed in half precision (FP16 or BF16) for speed, while certain operations that require numerical stability (such as loss accumulation and weight updates) are performed in full precision (FP32). ^[8]

A key technique in mixed precision training is loss scaling. Because gradients in deep networks can be very small (below $10^{-10}$ in some cases), converting them to FP16 can cause underflow, meaning small values get rounded to zero and gradient information is lost. ^[8] Loss scaling addresses this by multiplying the scalar loss value by a large scalar factor (for example, $2^{16}$) before backpropagation. Because the loss is a scalar, scaling it is computationally cheap and, by the chain rule, all downstream gradients are scaled by the same factor. After the backward pass, the gradients are divided by the scaling factor before the weight update.

Dynamic loss scaling adjusts the scaling factor automatically during training. It starts with a large scale factor and monitors for gradient overflow (NaN or Inf values). If overflow is detected, the weight update is skipped and the scale factor is reduced. If training proceeds without overflow for a set number of iterations, the scale factor is increased. This approach lets training use the highest possible precision without manual tuning. ^[8]

Mixed precision training typically achieves 1.5x to 3x speedup on modern GPUs with minimal impact on model accuracy. ^[8]

What is a scalar in physics?

In physics, a scalar is a physical quantity that is fully described by its magnitude alone, without any directional component. Scalars remain unchanged under coordinate transformations such as rotations and reflections, which makes them coordinate-invariant.

Examples of scalar quantities in physics:

Scroll sideways for more →

Scalar fields play an important role in modern physics. In quantum field theory, a scalar field is associated with spin-0 particles. The Higgs field, which gives mass to elementary particles through the Higgs mechanism, is a scalar field. The discovery of the Higgs boson at CERN in 2012 confirmed the existence of a fundamental scalar field in nature.

How is a scalar different from a vector, matrix, or tensor?

The following table compares scalars with other mathematical objects commonly used in machine learning: ^[1][7]

Scroll sideways for more →

In short, the difference is the number of axes: a scalar has zero axes (one number), a vector has one axis (an ordered list of numbers), a matrix has two axes (a grid of numbers), and a general tensor has any number of axes. ^[1][16] Every vector, matrix, and tensor is ultimately an array of scalars.

What is the history of the scalar concept?

The mathematical concept of a scalar has evolved over centuries alongside the development of algebra and geometry.

Francois Viete used the Latin term scalaris in 1591 to describe magnitudes that "ascend or descend proportionally" along a scale. However, the modern mathematical meaning took shape in the 19th century with the development of quaternion algebra. William Rowan Hamilton, who invented quaternions in 1843, used the term "scalar" in 1846 to refer to the real part of a quaternion, distinguishing it from the "vector" part that carried directional information. ^[3] Hamilton's usage established the scalar-vector distinction that persists in mathematics and physics today.

The formalization of vector spaces by Giuseppe Peano in 1888 and the subsequent axiomatization of abstract algebra in the early 20th century gave the term "scalar" its modern definition as an element of an arbitrary field. ^[10] This abstraction allowed mathematicians to work with scalars that are not ordinary real numbers, such as complex numbers, finite field elements, or even more exotic algebraic structures.

In the context of machine learning, the systematic use of scalar parameters dates back to the earliest neural network models. Frank Rosenblatt's perceptron (1958) used scalar weights to classify inputs, and the development of backpropagation by Rumelhart, Hinton, and Williams in 1986 formalized how scalar gradients could be propagated through multi-layer networks to update scalar weights efficiently. ^[5][6]

References

1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*. MIT Press. Chapter 2: Linear Algebra. https://www.deeplearningbook.org/contents/linear_algebra.html ↩
2. Strang, G. (2016). *Introduction to Linear Algebra* (5th ed.). Wellesley-Cambridge Press. ↩
3. Hamilton, W. R. (1846). "On Quaternions; or on a New System of Imaginaries in Algebra." *Philosophical Magazine*, 29(supplement), pp. 26-31. ↩
4. Halmos, P. R. (1958). *Finite-Dimensional Vector Spaces* (2nd ed.). Springer.
5. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." *Nature*, 323(6088), pp. 533-536. ↩
6. Rosenblatt, F. (1958). "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain." *Psychological Review*, 65(6), pp. 386-408. ↩
7. Kolda, T. G., & Bader, B. W. (2009). "Tensor Decompositions and Applications." *SIAM Review*, 51(3), pp. 455-500. ↩
8. Micikevicius, P., et al. (2018). "Mixed Precision Training." *International Conference on Learning Representations (ICLR)*. https://arxiv.org/abs/1710.03740 ↩
9. Jacob, B., et al. (2018). "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference." *IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 2704-2713. ↩
10. Peano, G. (1888). *Calcolo Geometrico secondo l'Ausdehnungslehre di H. Grassmann*. Turin: Fratelli Bocca Editori. ↩
11. Horn, R. A., & Johnson, C. R. (2012). *Matrix Analysis* (2nd ed.). Cambridge University Press.
12. Axler, S. (2015). *Linear Algebra Done Right* (3rd ed.). Springer. ↩
13. Paszke, A., et al. (2019). "PyTorch: An Imperative Style, High-Performance Deep Learning Library." *Advances in Neural Information Processing Systems (NeurIPS)*, 32. PyTorch documentation: https://docs.pytorch.org/docs/stable/tensors.html ↩
14. Kingma, D. P., & Ba, J. (2015). "Adam: A Method for Stochastic Optimization." *International Conference on Learning Representations (ICLR)*. ↩
15. NumPy Developers. "The N-dimensional array (ndarray)." NumPy Reference. https://numpy.org/doc/stable/reference/arrays.ndarray.html ↩
16. TensorFlow Developers. "Introduction to Tensors." TensorFlow Core Guide. https://www.tensorflow.org/guide/tensor ↩