Example

In machine learning, an example is a single data point that a model trains on or makes a prediction about: in supervised settings it is a pair $(x, y)$ where $x$ is a vector of features and $y$ is a label, and in unsupervised settings it is just the feature vector $x$. The word is used almost interchangeably with instance, sample, observation, record, row, and data point; different research communities prefer different terms, but the underlying object is the same: one row of data that the learning algorithm consumes. The choice of which examples to collect, how to label them, how to split them across training and evaluation, and how to weight them during training shapes almost every property of the resulting model. Examples are also the unit in which modern training scale is measured: the GPT-3 language model described by Brown and colleagues was a 175 billion parameter network trained on hundreds of billions of next-token prediction examples ^[7].

Explain like I'm 5 (ELI5)

Imagine you are teaching a friend to tell apples from oranges. Each time you hold up a piece of fruit and say what it is, that is one example. The fruit itself is what your friend looks at. The name ("apple" or "orange") is the answer. After enough examples your friend can guess on their own. A computer learns the same way, just with millions of examples instead of a handful. If you only ever show your friend yellow apples, they will struggle when a green apple shows up. The mix of examples you choose decides what the learner can and cannot do later.

What is the formal definition of an example?

In the standard formulation of supervised learning, examples are assumed to be drawn independently and identically distributed (iid) from a fixed but unknown joint distribution $P(X, Y)$ over an input space $\mathcal{X}$ and an output space $\mathcal{Y}$. A training data set of size $n$ is then

$$D = {(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)}$$

where each $(x_i, y_i)$ is one example. The features $x_i$ may be a real-valued vector, a sequence of tokens, a tensor of pixel intensities, a graph, or any other structured object. The label $y_i$ may be a class index for classification, a real number for regression, a sequence for sequence-to-sequence tasks, or even another structured object. Tom Mitchell's standard textbook frames the whole learning problem around this unit, defining a learner as a program that improves at a task "with experience" where the experience is a set of training examples ^[13].

In unsupervised settings the examples are just ${x_1, \ldots, x_n}$ with no labels attached. The iid assumption is convenient for theory but rarely holds exactly in practice. Real datasets tend to be biased by the collection process, by the population that was sampled, and by the labelers who annotated them.

Synonyms and terminology

Different communities use different words for the same idea, which causes a lot of needless confusion when reading across fields.

Scroll sideways for more →

"Sample" is the most slippery of these. A deep learning practitioner who says "the model saw 10 million samples" means 10 million examples. A classical statistician who says "we drew a sample of size 1000" means a single dataset of 1000 observations.

How does an example differ across learning paradigms?

The shape of an example depends on the kind of learning being done.

Scroll sideways for more →

In supervised learning the examples are the raw material. In self-supervised learning, examples are constructed from unlabeled text or images by hiding part of the input and asking the model to predict it. The trillions of next-token prediction examples used to pretrain modern large language models all come from this trick. In reinforcement learning, the unit is a transition or full trajectory, which is a sequence of state, action, and reward triples rather than a single (input, label) pair.

Labeled and unlabeled examples

The distinction between labeled and unlabeled examples sits at the heart of semi-supervised learning, which was given its modern formulation in the 2006 MIT Press volume edited by Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien ^[1]. Labels are usually expensive: a doctor must read the X-ray, a translator must produce the target sentence, a judge must rate the search result. Unlabeled inputs are usually cheap: web pages, photographs, and recordings can be collected by the billion at almost no cost.

Scroll sideways for more →

The ratio of labeled to unlabeled examples in a project drives many design choices. If labels are abundant, plain supervised learning is usually the right tool. If labels are scarce but inputs are plentiful, semi-supervised learning, self-training, and pretraining followed by fine-tuning all become attractive.

How do examples differ from instances?

"Example" and "instance" are usually treated as synonyms. The two communities that introduced them simply chose different words. There is one place where the distinction matters: multi-instance learning (MIL). MIL was introduced by Thomas Dietterich, Richard Lathrop, and Tomás Lozano-Pérez in a 1997 paper on drug activity prediction ^[2]. In MIL the unit of training is a bag of instances rather than a single instance. The label is attached to the bag. A drug molecule can fold into many shapes (conformers), and the molecule binds to a target if at least one of its shapes binds. Only the molecule's overall activity is observed, not which conformer is responsible. So the bag of conformers is the example, but each conformer is an instance. The paper defined the problem precisely: in the multiple instance setting "each example object may have many alternative feature vectors (instances) that describe it," yet only one may be responsible for the observed label, and their best axis-parallel rectangle method reached 89 percent correct predictions on the musk binding task ^[2].

This bag-versus-instance split shows up again in computer vision (where an image is a bag of patches), in document classification (where a document is a bag of sentences), and in pathology (where a slide is a bag of tiles). Outside of MIL the distinction collapses and the two words are interchangeable.

Batches of examples

Modern training rarely processes one example at a time. Instead, examples are grouped into a batch (sometimes called a mini-batch) and passed through the model together. The gradient of the loss is then averaged across the batch before the weights are updated.

Scroll sideways for more →

Batch size is one of the most discussed hyperparameters in deep learning. Small batches give noisier gradient estimates but often generalize better. Large batches use hardware more efficiently and reduce per-example variance, at the cost of needing more careful learning-rate tuning.

Train, validation, and test examples

A single dataset is almost always split into disjoint subsets so that performance can be measured honestly.

Scroll sideways for more →

A cardinal rule of supervised ML is that test examples must never be used to train or tune the model. When they are, the test error is no longer an honest estimate of how the model will perform on new data. The Elements of Statistical Learning describes this discipline as splitting the data into a training set, a validation set for model selection, and a test set used only once for the final assessment of the chosen model ^[14]. Information leakage between splits is one of the most common reasons published results fail to reproduce.

What makes an example hard or easy?

Not every example is equally informative. Some are easy: the model gets them right almost from the first epoch. Others are hard: the model misclassifies them again and again, or the loss on them stays high. The recognition that this matters led to several lines of work.

Curriculum learning, introduced by Yoshua Bengio and colleagues at ICML 2009, presents examples in order of increasing difficulty rather than at random ^[6]. The intuition is borrowed from human and animal learning: the paper notes that "humans and animals learn much better when the examples are not randomly presented but organized in a meaningful order which illustrates gradually more concepts" ^[6]. Bengio et al. reported faster convergence and better final solutions on shape recognition and language modeling tasks.

Hard-example mining goes the other way. Object detectors and face recognizers often train on a stream of mostly easy negatives. Online hard example mining (OHEM), proposed by Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick at CVPR 2016, deliberately oversamples the examples on which the current model performs worst ^[10]. This focuses gradient signal where it matters and tends to improve detection accuracy.

Noisy or mislabeled examples are a third category. They are hard for the wrong reason. A line of confident learning research led by Curtis Northcutt and colleagues at MIT audits popular benchmarks and finds that test sets like ImageNet, MNIST, and CIFAR contain real labeling errors that depress reported accuracies of strong models. Their 2021 study estimated an average of 3.4 percent label errors across ten widely used test sets and found 2,916 errors, about 6 percent, in the ImageNet validation set alone ^[11].

What are adversarial examples?

Adversarial examples are inputs that have been intentionally perturbed to make a model misclassify them, often by a margin invisible to humans. The phenomenon was reported by Christian Szegedy and colleagues in the paper Intriguing Properties of Neural Networks, posted in late 2013 and presented at ICLR 2014 ^[3]. They showed that almost any image could be perturbed by a tiny amount and cause a confident image classifier to assign the wrong label, and that the same adversarial examples transfer between models trained on different subsets of the data, which they argued exposes "fundamental blind-spots in our training algorithms" ^[3].

Ian Goodfellow, Jonathon Shlens, and Christian Szegedy followed up in the ICLR 2015 paper Explaining and Harnessing Adversarial Examples, which introduced the fast gradient sign method (FGSM) and argued that the underlying cause was the near-linear behavior of deep networks in high-dimensional input space rather than overfitting ^[4]. FGSM constructs an adversarial example as $x' = x + \varepsilon \cdot \text{sign}(\nabla_x J(\theta, x, y))$, where $\varepsilon$ controls the size of the perturbation and $J$ is the loss. Including these perturbed examples in training, called adversarial training, became one of the standard defenses.

Adversarial examples are a serious safety concern in any deployment where an attacker can craft inputs: spam filtering, malware detection, face recognition, and autonomous driving have all been shown to be vulnerable.

What is a counterfactual example?

A counterfactual example is closely related but built for a different purpose: explanation rather than attack. Sandra Wachter, Brent Mittelstadt, and Chris Russell argued in their 2017 paper Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR that data subjects have a legitimate interest in knowing what minimal change to their input would have flipped a model's decision ^[5]. If a loan application was denied, a useful explanation might be "if your annual income had been 4,000 USD higher, the model would have approved you." The counterfactual is an example that the model classifies the way the user wanted, and that is as close as possible to their actual input. Counterfactual explanations have since become a standard tool in the interpretable machine learning literature and are often discussed in the context of the EU General Data Protection Regulation.

How are synthetic examples generated?

When real labeled examples are scarce, synthetic ones can fill the gap. The simplest form is data augmentation: cropping, flipping, and color-jittering images, or paraphrasing text. More elaborate methods generate entirely new examples.

Scroll sideways for more →

The LLM-generated case has exploded in importance since 2022. Yizhong Wang and colleagues introduced Self-Instruct, a framework that bootstraps an instruction-tuning dataset by prompting a base model to generate new tasks from a small seed set ^[8]. Starting from just 175 human-written seed tasks, the method produced about 52,000 instructions paired with about 82,000 instances, then used them to fine-tune GPT-3. The authors reported a "33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT001," with only "a 5% absolute gap behind InstructGPT001" on expert-written novel tasks, despite using almost no human-written demonstrations beyond the seed ^[8]. Stanford Alpaca followed the same recipe with examples generated by GPT-3.5, and Anthropic's Constitutional AI used a model to critique and revise its own outputs to produce preference examples without human labelers in the loop.

Synthetic examples are not free. Errors compound: if the generator hallucinates a fact, every example built on it is wrong. The Self-Instruct authors themselves audited 200 random generations and found that a substantial fraction contained problems, so quality filtering and careful prompt design are the main defenses ^[8].

How do examples work in modern LLM training?

Large language models stretch the meaning of "example" in interesting ways. The same sequence of tokens can serve as many different examples depending on which sub-sequence is treated as the target.

Scroll sideways for more →

Long Ouyang and colleagues at OpenAI laid out this multi-stage pipeline in Training Language Models to Follow Instructions With Human Feedback, the 2022 InstructGPT paper ^[9]. Reinforcement learning from human feedback (RLHF) reframes "example" as a preference pair: the labeler picks the better of two model outputs, and the reward model learns to predict which a human would prefer. This pairwise structure traces back to the Bradley-Terry model from 1952 and turns subjective judgments into a differentiable training signal.

Few-shot learning and zero-shot learning further blur the line between training and inference. In Tom Brown and colleagues' 2020 paper Language Models Are Few-Shot Learners, the 175 billion parameter GPT-3 is shown $K$ input-output demonstrations inside the prompt and asked to complete the next case, "without any gradient updates or fine-tuning" ^[7]. The demonstrations act as examples, but they live in the context window rather than in a training file. The phrase "shot" comes from one-shot and few-shot learning in computer vision.

Sampling weights

Not every example contributes equally to the loss. A common pattern is to upweight rare classes or important subgroups so that they are not drowned out by the majority. In the simplest form the loss becomes

$$L = \frac{1}{n} \sum_{i=1}^{n} w_i \cdot \ell(f(x_i), y_i)$$

where $w_i$ is a per-example weight. Importance sampling, class-balanced loss, and curriculum reweighting are all variants of this idea. Choosing weights carelessly can introduce bias; choosing them deliberately is one of the main levers for handling class imbalance and distribution shift.

What formats are used to store examples?

Examples need to live somewhere. The format depends on scale, on access patterns, and on the framework in use.

Scroll sideways for more →

A TensorFlow tf.train.Example is, despite the name, exactly the data structure described in this article: a single instance, encoded as a dictionary of feature names to values ^[12]. The record format, TFRecord, just packs many of these protos into a binary stream that data loaders can read sequentially ^[12].

Data quality and example value

An example is only as good as its label. Mislabeled examples poison gradient signal, mislead reward models, and inflate or deflate evaluation metrics depending on which way the noise runs. The Northcutt confident learning audits found error rates of several percent in widely used benchmarks, averaging 3.4 percent across ten datasets ^[11]. Anthropic, OpenAI, and others have invested heavily in labeler training and inter-annotator agreement for the same reason. The trend in 2024 and 2025 has been toward fewer, higher-quality examples for fine-tuning, with synthetic generation used to amplify a small carefully curated seed rather than to flood the model with cheap data.

In pretraining the picture is different. Quantity still matters, but so does deduplication, contamination filtering, and quality scoring. The same web page repeated a thousand times in the corpus does not produce a thousand distinct examples worth of learning signal.

See also

• Instance
• Training data
• Labeled data
• Unlabeled data
• Feature
• Label
• Dataset
• Supervised learning
• Unsupervised learning
• Semi-supervised learning
• In-context learning
• Few-shot learning
• Zero-shot learning
• Active learning
• Batch

References

1. Chapelle, O., Schölkopf, B., and Zien, A. (eds). *Semi-Supervised Learning*. MIT Press, 2006. https://mitpress.mit.edu/9780262514125/semi-supervised-learning/ ↩
2. Dietterich, T. G., Lathrop, R. H., and Lozano-Pérez, T. "Solving the multiple instance problem with axis-parallel rectangles." *Artificial Intelligence* 89, 1997, pp. 31-71. https://doi.org/10.1016/S0004-3702(96)00034-3 ↩
3. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. "Intriguing properties of neural networks." ICLR 2014. https://arxiv.org/abs/1312.6199 ↩
4. Goodfellow, I. J., Shlens, J., and Szegedy, C. "Explaining and Harnessing Adversarial Examples." ICLR 2015. https://arxiv.org/abs/1412.6572 ↩
5. Wachter, S., Mittelstadt, B., and Russell, C. "Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR." *Harvard Journal of Law & Technology* 31, 2017. https://arxiv.org/abs/1711.00399 ↩
6. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. "Curriculum Learning." ICML 2009, pp. 41-48. https://ronan.collobert.com/pub/2009_curriculum_icml.pdf ↩
7. Brown, T. B., et al. "Language Models are Few-Shot Learners." NeurIPS 2020. https://arxiv.org/abs/2005.14165 ↩
8. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. "Self-Instruct: Aligning Language Models with Self-Generated Instructions." ACL 2023. https://arxiv.org/abs/2212.10560 ↩
9. Ouyang, L., et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS 2022. https://arxiv.org/abs/2203.02155 ↩
10. Shrivastava, A., Gupta, A., and Girshick, R. "Training Region-Based Object Detectors with Online Hard Example Mining." CVPR 2016. https://arxiv.org/abs/1604.03540 ↩
11. Northcutt, C. G., Athalye, A., and Mueller, J. "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks." NeurIPS 2021 Datasets and Benchmarks Track. https://arxiv.org/abs/2103.14749 ↩
12. TensorFlow. "TFRecord and tf.train.Example." https://www.tensorflow.org/tutorials/load_data/tfrecord ↩
13. Mitchell, T. M. *Machine Learning*. McGraw-Hill, 1997. ↩
14. Hastie, T., Tibshirani, R., and Friedman, J. *The Elements of Statistical Learning*. Springer, 2nd edition, 2009. ↩