# Pre-training

> Source: https://aiwiki.ai/wiki/pre-training
> Updated: 2026-07-31
> Fact-checked: 2026-07-31
> Categories: Artificial Intelligence, Computer Vision, Deep Learning, Machine Learning, Natural Language Processing, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Pre-training." aiwiki.ai, 31 Jul 2026. https://aiwiki.ai/wiki/pre-training
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Pre-training** is a stage of [machine learning](https://aiwiki.ai/wiki/machine_learning) in which a model learns parameters from a source dataset or source objective before those parameters are reused or adapted for a target use. The source stage can be supervised, self-supervised, or, less commonly, unsupervised in the narrower probabilistic sense. The resulting pre-trained model may be used as a fixed feature extractor, fine-tuned in full, adapted through a smaller set of trainable parameters, prompted without parameter updates, or subjected to further pre-training on another corpus.[1][3][9][10][12][22][23][24]

Pre-training is therefore broader than any one recipe. It does not necessarily use web data, transformers, billions of parameters, or self-supervision. It also does not make every pre-trained model a [foundation model](https://aiwiki.ai/wiki/foundation_models). The foundation-model report introduced that term for models trained on broad data, generally with self-supervision at scale, that can be adapted to a wide range of downstream tasks. A small image classifier pre-trained on labeled ImageNet data is a pre-trained model, but it need not meet that broader description.[3][13]

The practical purpose is transfer. Instead of estimating every parameter from a target dataset alone, a practitioner starts from representations learned elsewhere. Whether this helps depends on the source data, objective, architecture, target task, and adaptation method. Pre-training can reduce the amount of labeled target data needed in a particular experiment, but it does not guarantee a fixed improvement or eliminate the need for task-specific evaluation.[4][20][21][22]

## Definition and scope

The word "pre-training" is relational: a training phase is "pre" with respect to later use. A model can undergo an initial broad pre-training run, continued pre-training on data from a particular domain, and then supervised fine-tuning. Research on domain-adaptive and task-adaptive pre-training has used "pre-training" for this second source-stage optimization even though it occurs after an earlier pre-training run.[24]

Several neighboring terms describe different parts of the process:

- **[Transfer learning](https://aiwiki.ai/wiki/transfer_learning)** is the broader reuse of information learned in one setting to improve learning or performance in another. Pre-training creates parameters or representations that can be transferred.
- **[Fine-tuning](https://aiwiki.ai/wiki/fine_tuning)** continues optimization of some or all model parameters for a target task or data distribution. It is one adaptation method, not part of the definition of pre-training.
- **[Self-supervised learning](https://aiwiki.ai/wiki/self-supervised_learning)** constructs supervision from the data itself, such as hidden tokens, future tokens, paired views, or masked image patches. It is common in modern pre-training but is not required.
- **Supervised pre-training** uses externally supplied labels during the source stage. Large labeled image datasets made this approach important in [computer vision](https://aiwiki.ai/wiki/computer_vision).[2][3]
- **Post-training** usually refers to later procedures that shape a model's behavior after broad pre-training, such as instruction tuning or preference optimization. These procedures should not be treated as pre-training objectives merely because they continue to update the same model.

Pre-training also differs from training from scratch. "From scratch" normally means that the target optimization begins from a random or otherwise non-learned initialization. A pre-trained initialization already reflects a source dataset and objective. That inheritance is useful when source and target are compatible, but it can also carry source-specific errors, biases, memorized sequences, and design constraints into the target system.[4][13][29][31][32]

## Historical development

### Layer-wise initialization and supervised vision

In 2006, Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh described a greedy learning algorithm that constructed a deep belief network one layer at a time. Their unsupervised procedure initialized a slower fine-tuning stage. The work demonstrated a specific way to train deep probabilistic models; it is more precise to describe it as an important early pre-training method than as a general solution to every optimization problem in deep networks.[1]

Large labeled visual datasets supported a different route. The original ImageNet paper described a hierarchy intended to contain hundreds of images for tens of thousands of WordNet synsets and reported 3.2 million images in the portion then completed.[2] AlexNet was trained with supervision for the 2012 ImageNet competition and achieved a 15.3 percent top-5 test error for its seven-model ensemble, compared with 26.2 percent for the second-place entry. The AlexNet paper explicitly said that its main experiments did not use unsupervised pre-training.[3]

AlexNet's competition result and ImageNet transfer should not be collapsed into one claim. Subsequent work studied when features learned by an ImageNet classifier could transfer to other tasks. Yosinski and colleagues found that transferability varied by layer and by the distance between source and target tasks; in their experiments, transferred initialization could still outperform random initialization even when tasks were distant.[4]

### Word and contextual representations

In [natural language processing](https://aiwiki.ai/wiki/natural_language_processing), Word2Vec introduced the continuous bag-of-words and skip-gram architectures in 2013. The former predicted a current word from its context, while the latter predicted surrounding words from a current word.[5] GloVe followed with a weighted log-bilinear model built from global word-word co-occurrence statistics.[6] These methods learned one principal vector per word or token type.

ELMo moved from static vectors to context-dependent representations. Its bidirectional language model used character convolutions and stacked LSTMs, and downstream systems learned a task-specific combination of representations from different layers.[7] The same surface word could therefore receive a different representation in a different sentence.

### Transformers and general-purpose language models

The 2017 Transformer paper proposed an encoder-decoder architecture based on attention, dispensing with recurrence and convolution in the sequence transduction model it studied.[8] The architecture itself was not a pre-training objective. Its parallel sequence processing and flexible attention blocks nevertheless became a common basis for later pre-trained models.

OpenAI's 2018 generative pre-training work combined an autoregressive language-model objective on unlabeled text with supervised fine-tuning for several target tasks.[9] BERT instead pre-trained a bidirectional Transformer encoder with masked language modeling and next-sentence prediction, then fine-tuned the resulting parameters for tasks including question answering and natural-language inference.[10] T5 later compared architectures, objectives, datasets, and transfer methods in a unified text-to-text framework.[11]

GPT-3 scaled an autoregressive language model to 175 billion parameters and evaluated zero-shot, one-shot, and few-shot use without gradient updates for each task.[12] That paper provided evidence for task-agnostic pre-training and in-context task specification, not proof that scaling always yields a particular capability at a predictable parameter count.

## Pre-training objectives

An objective defines the errors whose gradients update the model. Different objectives expose different information and impose different constraints. Three widely used families are autoregressive prediction, masked reconstruction, and contrastive learning.

### Autoregressive prediction

For a token sequence $$x_1,\ldots,x_T$$, a left-to-right [autoregressive model](https://aiwiki.ai/wiki/autoregressive_model) minimizes the negative log-likelihood

$$
\mathcal{L}_{\mathrm{AR}}(\theta)
=-\sum_{t=1}^{T}\log p_{\theta}(x_t\mid x_{<t}).
$$

Each observed prefix supplies a prediction target for its next token. The GPT line of work uses this factorization.[9][12] Autoregressive pre-training supports sequence generation because the learned conditional distribution can be sampled one token at a time. It does not by itself specify the data mixture, tokenizer, architecture size, decoding rule, or later behavioral tuning.

### Masked prediction and reconstruction

A [masked language model](https://aiwiki.ai/wiki/masked_language_model) selects positions $$M$$ and predicts the hidden tokens from the visible context:

$$
\mathcal{L}_{\mathrm{MLM}}(\theta)
=-\sum_{i\in M}\log p_{\theta}(x_i\mid x_{\setminus M}).
$$

BERT selected 15 percent of token positions. Of those selected positions, 80 percent were replaced by `[MASK]`, 10 percent by a random token, and 10 percent were left unchanged.[10] These numbers describe BERT's implementation, not the definition of masked modeling.

Later methods changed the corruption or prediction task. RoBERTa used dynamic masking, removed the next-sentence-prediction objective, trained longer, and used more data in its study.[14] SpanBERT masked contiguous spans and added a span-boundary objective.[15] ELECTRA trained a discriminator to identify tokens replaced by samples from a generator rather than predicting only the original identities of masked tokens.[16]

Masked autoencoders apply a related idea to images. The MAE paper masked a large random subset of image patches, using 75 percent as its default high masking ratio, and trained a decoder to reconstruct pixels for the missing patches.[19]

### Contrastive and paired-data objectives

[Contrastive learning](https://aiwiki.ai/wiki/contrastive_learning) trains representations so that designated positive pairs receive more similar embeddings than competing pairs. The definition of a positive pair is part of the method. SimCLR treated two independently augmented views of the same image as a positive pair and found that augmentation composition, a projection head, and batch size materially affected results in its experiments.[17]

CLIP used paired images and natural-language text rather than two views of one image. It trained image and text encoders on 400 million collected image-text pairs and used the learned text encoder to construct zero-shot classifiers from class names or descriptions.[18] This is contrastive multimodal pre-training with noisy natural-language supervision, not an example of training without any supervision signal.

| Objective family | Training signal | Representative examples | Main qualification |
|---|---|---|---|
| Autoregressive prediction | A preceding sequence predicts the next element | GPT and other causal language models | The left-to-right factorization is not inherently bidirectional |
| Masked prediction or reconstruction | Visible context predicts hidden tokens, spans, audio units, or image patches | BERT, SpanBERT, MAE | Masking rate and corruption process are method choices |
| Replaced-token detection | A discriminator identifies generated substitutions | ELECTRA | The generator is discarded after pre-training in the original method |
| Contrastive learning | Positive pairs are distinguished from alternatives in representation space | SimCLR, CLIP, wav2vec 2.0 | Results depend on how positives, negatives, and augmentations are constructed |
| Supervised source-task training | Human or curated labels define the source task | ImageNet classification | The source label space can encourage task-specific features |

## Training workflow

There is no single universal pre-training pipeline, but a defensible workflow separates decisions that are often hidden by the phrase "trained on a large dataset."

1. **Define the intended transfer.** Specify the modalities, target settings, acceptable failure modes, and whether the model will be frozen, fine-tuned, prompted, or continually updated. An objective that is useful for one transfer path may be poorly matched to another.

2. **Collect and document data.** Record sources, licenses or terms, collection dates, languages, sampling rules, and exclusions. Dataset size must be reported with its unit. Bytes, documents, images, pairs, words, and post-[tokenization](https://aiwiki.ai/wiki/tokenization) tokens are not interchangeable measures.

3. **Filter, deduplicate, and partition.** Filtering can remove low-quality or unsafe material, but it can also remove dialects or topics unevenly. Deduplication can reduce repeated examples, memorization, and train-test overlap. Validation and test sets must be checked against the pre-training corpus when contamination would invalidate evaluation.[29][30]

4. **Choose representations and an objective.** Text systems select a tokenizer and context length; image systems select resolution, patches, crops, or augmentations; audio systems select waveform or learned latent representations. These choices determine what information the network sees.

5. **Optimize and checkpoint.** Architecture, optimizer, numerical precision, batch construction, parallelism, and learning-rate schedule interact. Loss curves and held-out evaluations can detect divergence or [overfitting](https://aiwiki.ai/wiki/overfitting), but low source loss does not establish downstream usefulness or safety.

6. **Evaluate transfer.** Compare against training from scratch and against relevant pre-trained baselines under the same target-data and tuning budget. Report the adaptation method, not merely the name of the source model.

7. **Document the release.** A useful model card or technical report identifies the base and adapted variants, data scope, intended use, evaluation conditions, compute reporting method, and known limitations. Releasing weights without those distinctions makes later claims difficult to verify.

### Documented examples

| System or study | Source signal and data | What was transferred or evaluated |
|---|---|---|
| Deep belief network, 2006 | Layer-wise unsupervised learning followed by fine-tuning on MNIST | Initialization of a deep generative model and classifier[1] |
| AlexNet, 2012 | Supervised ImageNet classification | Competition classification; later studies examined transfer of learned layers[3][4] |
| Word2Vec, 2013 | Local text-context prediction | Static word vectors evaluated on syntactic and semantic relationships[5] |
| BERT, 2018 | Masked tokens and next-sentence prediction on BooksCorpus and English Wikipedia | Fine-tuning on sentence, token, and question-answering tasks[10] |
| SimCLR, 2020 | Two augmented views of each ImageNet image | Visual representations evaluated with linear classification and fine-tuning[17] |
| wav2vec 2.0, 2020 | Contrastive prediction over quantized latent speech units from untranscribed audio | Speech recognition after fine-tuning with labeled audio, including low-resource settings[20] |
| CLIP, 2021 | 400 million collected image-text pairs | Zero-shot and transfer evaluation across visual tasks[18] |
| MAE, 2022 | Reconstruction of masked ImageNet patches | Fine-tuning and linear probing for image recognition tasks[19] |
| Llama 3.1 405B, 2024 | Autoregressive pre-training on 15.6 trillion text tokens in the technical paper | Pre-trained base model followed by distinct post-training stages[27] |

The table reports the units and evaluation described by each source. It deliberately does not compare "parameter count" with embedding dimension or dataset size, and it does not infer commercial training cost from a hardware count.

## Transfer and adaptation

Two basic adaptation strategies are feature extraction and full fine-tuning. In feature extraction, the source model remains frozen and a target model consumes its hidden representations. In full fine-tuning, the target loss updates the source parameters. A comparative NLP study found that neither method was uniformly superior across its tasks; performance depended on the model and the similarity between source and target tasks.[22]

Parameter-efficient methods occupy the space between those endpoints. Adapter modules insert small trainable components while keeping the original network fixed. In the original adapter study, the method came within 0.4 percentage points of full fine-tuning on the reported GLUE aggregate while adding 3.6 percent task-specific parameters. That result is specific to the study's BERT models, tasks, and implementation, rather than a general performance guarantee.[23]

Pre-training can improve sample efficiency under controlled conditions. ULMFiT reported that, on IMDb, a model using 100 labeled examples plus 50,000 unlabeled examples matched a from-scratch model trained with 100 times as much labeled data. Other tasks and settings in the same study produced different ratios.[21] The bounded result supports the value of transfer without justifying a universal claim that pre-training reduces labels by 10 to 100 times.

Continued pre-training is another adaptation path. The domain-adaptive pre-training study continued RoBERTa training on biomedical, computer-science, news, or review corpora and then on unlabeled task data. It found gains across the eight classification tasks studied, including after an earlier domain-adaptive phase.[24] This shows why "pre-training" need not be a single first stage, but it does not imply that continued training on any domain corpus will help.

## Scale, compute, and cost

Pre-training compute depends on the number of operations per example, the number of examples or tokens processed, and the hardware and software efficiency of the run. Parameter count alone is not a cost measure. Two models with the same number of parameters can use different token counts, sequence lengths, sparsity patterns, numerical formats, parallel layouts, and numbers of training steps.

Kaplan and colleagues measured approximate power-law relationships between language-model loss and model size, dataset size, and training compute over the ranges they studied.[25] Hoffmann and colleagues later fit a different compute-optimal allocation from experiments on more than 400 models. Their Chinchilla model used 70 billion parameters and 1.4 trillion tokens, compared with the larger 280-billion-parameter Gopher under a similar training-compute budget.[26] These are empirical scaling studies with stated model families and data ranges. Their fitted relationships are not laws that can be extrapolated without limit.

Meta's Llama 3 technical report documented a 405-billion-parameter model pre-trained on 15.6 trillion tokens using up to 16,000 H100 GPUs.[27] The accompanying Llama 3.1 model card reported 30.84 million H100-80GB GPU-hours for the 405B model and 39.3 million GPU-hours cumulatively across the 8B, 70B, and 405B collection.[28] GPU-hours are a hardware-time measure, not a disclosed cash cost. Converting them into money or energy requires assumptions about utilization, pricing, device power, cooling, datacenter overhead, and which failed or experimental runs are included.

For that reason, generalized tables assigning dollar costs to BERT, GPT-3, T5, CLIP, or Llama from public cloud prices are estimates, not primary measurements. A cost estimate can be useful if its hardware, utilization, time, price date, and coverage are explicit, but it should not be presented as the developer's reported expenditure.

## Data quality and evaluation

Dataset composition is a modeling choice. The Pile, for example, described an 825.18 GiB English corpus assembled from 22 constituent datasets.[38] LAION-5B described 5.85 billion CLIP-filtered image-text pairs, including 2.32 billion English pairs, and explicitly discussed the implications and flaws of collection at that scale.[39] DataComp provided a 12.8-billion-pair CommonPool as an index of image URLs and associated text for controlled dataset-filtering experiments.[40] These resources use different units and collection procedures, so their headline sizes cannot be compared as if they measured the same thing.

C4 illustrates why documentation matters. Its audit reported 365 million English documents, 156 billion SpaCy tokens, and 305 GB compressed. The authors found benchmark contamination and analyzed a blocklist filter that removed documents unevenly. Among documents assigned the highest dialect probability to African American English, 42 percent were excluded, compared with 6.2 percent among those assigned to White American English. Those percentages characterize one classifier-based audit of one filtering pipeline, not every web corpus or every text written in those dialects.[29]

Duplicate content creates another problem. In experiments on several language-model corpora, deduplication reduced the rate at which models emitted memorized text by about tenfold and reduced train-test overlap without worsening perplexity in the tested settings.[30] This is strong evidence for deduplication in those experiments, but the exact effect will vary with the corpus, model, and definition of duplication.

Evaluation contamination occurs when pre-training data contains benchmark examples or close variants. It can make a model appear to generalize when it has already encountered test material. Searches for exact strings, normalized matches, and near-duplicates can help, but they do not prove the absence of semantic or translated overlap. A credible evaluation reports what checks were possible and treats uncertain contamination as a limitation.[29][30]

## Applications and benefits

Pre-trained representations have been evaluated across language, vision, speech, and multimodal tasks:

- In language, GPT-style causal models support generation and in-context task specification, while BERT-style encoders can be fine-tuned for classification, token labeling, and extractive question answering.[9][10][12]
- In vision, ImageNet-trained classifiers supplied features and initializations for other visual tasks, while SimCLR and MAE studied self-supervised alternatives to label-based source training.[4][17][19]
- In speech, wav2vec 2.0 pre-trained on untranscribed audio and then fine-tuned for speech recognition. Its reported low-resource experiments included a model fine-tuned with ten minutes of labeled speech after large-scale unlabeled pre-training.[20]
- In multimodal learning, CLIP aligned image and text representations and evaluated zero-shot classifiers formed from text prompts.[18]

The main benefit is reuse: one source run can support more than one target model or evaluation. That can improve target performance, convergence, or label efficiency, as the cited studies demonstrate in particular settings. A frozen model can also avoid storing a complete fine-tuned copy for every task, while adapter-style methods limit the number of task-specific parameters.[22][23]

The benefit is conditional. Features often become more specialized in later layers, and source-target mismatch can reduce transfer quality.[4] A pre-trained model can be too large for the target latency or memory budget. An objective can omit information needed by the target. A strong result on one benchmark can disappear under distribution shift or a different adaptation budget.

## Risks and limitations

### Bias and representational harm

Pre-training learns statistical associations in its data. Caliskan, Bryson, and Narayanan showed that word embeddings reproduced several human-like associations measured with adaptations of implicit-association tests, including race- and gender-related effects.[32] A later study tested 11 word embeddings and two pre-trained language models and found significant negative associations involving language about people with disabilities across the models studied.[33] These results do not mean that every association has the same downstream effect, but they show that neutral optimization does not produce socially neutral representations.

Filtering can also introduce bias before optimization begins. The C4 blocklist result demonstrates that a rule intended to remove undesirable content can have unequal effects on dialect-classified documents.[29] Dataset audits should therefore examine both included and excluded material.

### Memorization and privacy

A model can reproduce sequences from its training data. Carlini and colleagues extracted hundreds of verbatim sequences from GPT-2, including personally identifying information in some cases, even when a sequence appeared in only one source document.[31] The result was an attack on a particular model family and should not be converted into a claim that every model exposes every record. It does establish that public collection and next-token prediction do not by themselves prevent memorization.

Deduplication, careful data selection, privacy testing, and restricted release can reduce some risks, but none is a universal proof against extraction. The risk also depends on repetition, model size, access mode, and the attacker's knowledge.[30][31]

### Compute, energy, and access

Large runs can require substantial accelerator time. Their environmental effect cannot be inferred from parameter count or GPU-hours alone. A measurement study found that carbon intensity varied by roughly five to ten times across geographic locations even within one organization, and that model, processor, datacenter, and location choices could produce much larger combined differences. The authors also warned that retroactive estimates are difficult when energy and infrastructure details were not recorded.[34]

Compute requirements can limit who can reproduce a run, explore alternatives, or audit data and training decisions. Sparse activation, parameter sharing, and more efficient kernels can reduce particular costs, but they do not automatically reduce total resource use if saved capacity is used to scale the model or dataset.

### Transparency and scope

Training data for many commercial models is only partly described. Missing provenance, filtering, tokenization, optimizer, and compute details make exact replication and independent cost analysis impossible. Model cards and technical reports can narrow that gap, but an independently verifiable article should distinguish disclosed values from estimates and avoid filling undisclosed fields with plausible-looking numbers.

A base model's capabilities also differ from those of a post-trained assistant. Instruction tuning, preference optimization, safety filters, retrieval, tools, and system prompts can materially change observed behavior. Evaluations of an adapted product should not be attributed solely to pre-training.

## Research directions

Current research changes different parts of the pipeline:

- **Data selection and documentation:** Dataset audits, deduplication, contamination checks, and DataComp-style controlled filtering experiments test whether better selected data can outperform merely larger collections.[29][30][40]
- **Compute allocation:** Scaling-law studies ask how parameter count and processed data should change under a fixed compute budget.[25][26]
- **Parameter sharing:** ALBERT factorized its embedding parameterization and shared parameters across layers. Its BERT-large-like configuration had 18 times fewer parameters and trained about 1.7 times faster in the reported setup.[35]
- **Sparse activation:** Switch Transformers routed each input through selected experts and demonstrated pre-training at up to a trillion parameters.[41] Mixtral 8x7B exposed 47 billion parameters while using about 13 billion active parameters per token in its architecture.[36] Sparse parameter count and active compute should therefore be reported separately.
- **Memory-aware kernels:** FlashAttention computed exact attention with tiling designed to reduce transfers between high-bandwidth memory and on-chip SRAM. Its paper reported different speedups for BERT, GPT-2, and long-sequence benchmarks, rather than one universal acceleration factor.[37]
- **Continued and multimodal pre-training:** Domain-adaptive language training, paired image-text learning, speech representation learning, and masked image modeling extend pre-training beyond a single text-only source run.[18][19][20][24]

These directions address different bottlenecks. Parameter-efficient fine-tuning reduces target-specific storage, while sparse experts and attention kernels alter source-stage computation. Quantization and knowledge distillation are often applied after a base model is trained. Grouping all of them under "faster pre-training" obscures what resource each method changes.

## Terminology

Both **pre-training** and **pretraining** appear in the research literature. This article uses the hyphenated form in prose and preserves the spelling used in paper titles. "Pre-trained" describes a model after the source stage.

The term does not identify a model's modality, objective, openness, data rights, scale, or safety properties. Those characteristics must be stated separately.

## See also

- [Transfer learning](https://aiwiki.ai/wiki/transfer_learning)
- [Fine-tuning](https://aiwiki.ai/wiki/fine_tuning)
- [Self-supervised learning](https://aiwiki.ai/wiki/self-supervised_learning)
- [Representation learning](https://aiwiki.ai/wiki/representation_learning)
- [Foundation models](https://aiwiki.ai/wiki/foundation_models)
- [Language model](https://aiwiki.ai/wiki/language_model)
- [Large language model](https://aiwiki.ai/wiki/large_language_model)
- [Masked language model](https://aiwiki.ai/wiki/masked_language_model)
- [Autoregressive model](https://aiwiki.ai/wiki/autoregressive_model)
- [Contrastive learning](https://aiwiki.ai/wiki/contrastive_learning)
- [Scaling laws](https://aiwiki.ai/wiki/scaling_laws)
- [Tokenization](https://aiwiki.ai/wiki/tokenization)

## References

1. Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh, "A Fast Learning Algorithm for Deep Belief Nets," Neural Computation, 2006. https://pubmed.ncbi.nlm.nih.gov/16764513/
2. Jia Deng et al., "ImageNet: A Large-Scale Hierarchical Image Database," CVPR, 2009. https://www.image-net.org/static_files/papers/imagenet_cvpr09.pdf
3. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," NeurIPS, 2012. https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
4. Jason Yosinski et al., "How Transferable Are Features in Deep Neural Networks?" NeurIPS, 2014. https://proceedings.neurips.cc/paper_files/paper/2014/hash/532a2f85b6977104bc93f8580abbb330-Abstract.html
5. Tomas Mikolov et al., "Efficient Estimation of Word Representations in Vector Space," 2013. https://arxiv.org/abs/1301.3781
6. Jeffrey Pennington, Richard Socher, and Christopher D. Manning, "GloVe: Global Vectors for Word Representation," EMNLP, 2014. https://aclanthology.org/D14-1162/
7. Matthew E. Peters et al., "Deep Contextualized Word Representations," NAACL, 2018. https://aclanthology.org/N18-1202/
8. Ashish Vaswani et al., "Attention Is All You Need," NeurIPS, 2017. https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
9. Alec Radford et al., "Improving Language Understanding by Generative Pre-Training," OpenAI, 2018. https://openai.com/index/language-unsupervised/
10. Jacob Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," NAACL, 2019. https://aclanthology.org/N19-1423/
11. Colin Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research, 2020. https://www.jmlr.org/papers/v21/20-074.html
12. Tom B. Brown et al., "Language Models Are Few-Shot Learners," NeurIPS, 2020. https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
13. Rishi Bommasani et al., "On the Opportunities and Risks of Foundation Models," 2021. https://arxiv.org/abs/2108.07258
14. Yinhan Liu et al., "RoBERTa: A Robustly Optimized BERT Pretraining Approach," 2019. https://arxiv.org/abs/1907.11692
15. Mandar Joshi et al., "SpanBERT: Improving Pre-training by Representing and Predicting Spans," Transactions of the Association for Computational Linguistics, 2020. https://aclanthology.org/2020.tacl-1.5/
16. Kevin Clark et al., "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators," ICLR, 2020. https://arxiv.org/abs/2003.10555
17. Ting Chen et al., "A Simple Framework for Contrastive Learning of Visual Representations," ICML, 2020. https://proceedings.mlr.press/v119/chen20j.html
18. Alec Radford et al., "Learning Transferable Visual Models From Natural Language Supervision," ICML, 2021. https://proceedings.mlr.press/v139/radford21a.html
19. Kaiming He et al., "Masked Autoencoders Are Scalable Vision Learners," CVPR, 2022. https://openaccess.thecvf.com/content/CVPR2022/html/He_Masked_Autoencoders_Are_Scalable_Vision_Learners_CVPR_2022_paper.html
20. Alexei Baevski et al., "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations," NeurIPS, 2020. https://proceedings.neurips.cc/paper_files/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html
21. Jeremy Howard and Sebastian Ruder, "Universal Language Model Fine-tuning for Text Classification," ACL, 2018. https://aclanthology.org/P18-1031/
22. Matthew E. Peters, Sebastian Ruder, and Noah A. Smith, "To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks," RepL4NLP, 2019. https://aclanthology.org/W19-4302/
23. Neil Houlsby et al., "Parameter-Efficient Transfer Learning for NLP," ICML, 2019. https://proceedings.mlr.press/v97/houlsby19a.html
24. Suchin Gururangan et al., "Don't Stop Pretraining: Adapt Language Models to Domains and Tasks," ACL, 2020. https://aclanthology.org/2020.acl-main.740/
25. Jared Kaplan et al., "Scaling Laws for Neural Language Models," 2020. https://arxiv.org/abs/2001.08361
26. Jordan Hoffmann et al., "Training Compute-Optimal Large Language Models," NeurIPS, 2022. https://arxiv.org/abs/2203.15556
27. Aaron Grattafiori et al., "The Llama 3 Herd of Models," Meta, 2024. https://ai.meta.com/research/publications/the-llama-3-herd-of-models/
28. Meta, "Llama 3.1 Model Card," 2024. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
29. Jesse Dodge et al., "Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus," EMNLP, 2021. https://aclanthology.org/2021.emnlp-main.98/
30. Katherine Lee et al., "Deduplicating Training Data Makes Language Models Better," ACL, 2022. https://aclanthology.org/2022.acl-long.577/
31. Nicholas Carlini et al., "Extracting Training Data from Large Language Models," USENIX Security, 2021. https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting
32. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan, "Semantics Derived Automatically from Language Corpora Contain Human-Like Biases," Science, 2017. https://pubmed.ncbi.nlm.nih.gov/28408601/
33. Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson, "A Study of Implicit Language Model Bias Against People With Disabilities," COLING, 2022. https://aclanthology.org/2022.coling-1.113/
34. David Patterson et al., "Carbon Emissions and Large Neural Network Training," 2021. https://arxiv.org/abs/2104.10350
35. Zhenzhong Lan et al., "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations," ICLR, 2020. https://arxiv.org/abs/1909.11942
36. Albert Q. Jiang et al., "Mixtral of Experts," 2024. https://arxiv.org/abs/2401.04088
37. Tri Dao et al., "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," NeurIPS, 2022. https://arxiv.org/abs/2205.14135
38. Leo Gao et al., "The Pile: An 800GB Dataset of Diverse Text for Language Modeling," 2020. https://arxiv.org/abs/2101.00027
39. Christoph Schuhmann et al., "LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models," NeurIPS Datasets and Benchmarks, 2022. https://arxiv.org/abs/2210.08402
40. Samir Yitzhak Gadre et al., "DataComp: In Search of the Next Generation of Multimodal Datasets," NeurIPS Datasets and Benchmarks, 2023. https://arxiv.org/abs/2304.14108
41. William Fedus, Barret Zoph, and Noam Shazeer, "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity," Journal of Machine Learning Research, 2022. https://arxiv.org/abs/2101.03961