Transfer Learning

Transfer learning is a family of machine learning methods in which information learned from one source domain or task is used while learning a target domain or task. In a widely used formalization, a domain consists of an input feature space and a probability distribution over that space, while a task consists of an output space and a predictive function. Transfer learning applies when the source and target domains differ, the source and target tasks differ, or both, and the source information is used to improve learning of the target predictor.^[1][2]

The transferred information can take several forms, including selected source examples, a learned representation, model parameters, or relationships among entities. A familiar modern workflow is to pretrain a model on a data-rich objective and then adapt it to a downstream task. Fixed feature extraction, partial or full fine-tuning, adapter modules, and low-rank updates are different adaptation strategies within that workflow. They do not guarantee an improvement. If reuse makes a defined target result worse than an appropriate non-transfer baseline, the result is negative transfer.^[1][2][3]

Transfer learning is therefore a problem setting and a collection of methods, not one algorithm. The effect of a transfer depends on the source data, source objective, target data, target metric, model, adaptation procedure, and evaluation distribution. Evidence from one source-target pair does not establish that the same choice will work for another pair.

Problem setting

Let the source be a domain and task from which information is available, and let the target be the domain and task on which performance is required. The distinction between a domain change and a task change is important:

Scroll sideways for more →

Pan and Yang's 2010 survey organized transfer learning into three settings. In inductive transfer learning, the target task differs from the source task and labeled target data are available. In transductive transfer learning, the task is the same but the domains differ, with labeled source data and no labeled target data in the survey's formulation. The latter includes much of what is called domain adaptation. In unsupervised transfer learning, the target objective is unsupervised and neither domain provides labels for that objective.^[1] Later surveys use overlapping but not always identical taxonomies, so a study should state its data and label assumptions rather than rely on a category name alone.^[2]

The same formal setting can involve one source or several sources. It can also involve a source and target with the same input representation but different distributions, or heterogeneous input spaces that require an additional mapping. These are materially different problems. A method that assumes a shared feature space cannot be transferred unchanged to a heterogeneous setting.

What can be transferred

Two broad questions are what information is reused and how that information enters target learning.^[1][2]

• Instance transfer reuses or reweights source examples. The weights are intended to emphasize source observations that are relevant to the target and reduce the influence of mismatched observations.
• Representation transfer learns features that are reused by a target model. The representation can be frozen, combined with target-specific features, or updated during adaptation.
• Parameter transfer initializes or constrains the target model using source parameters, shared priors, or a structured update to those parameters.
• Relational transfer reuses relationships or structure among entities when source and target domains have compatible relational patterns.

These mechanisms can be combined. For example, a pretrained network supplies both an initialization and a representation, while fine-tuning updates some or all of that representation with target examples. Calling the result "parameter transfer" or "representation transfer" highlights a mechanism, not an exclusive category.

Historical development

Work on transferring learned neural-network parameters predates the large pretrained models associated with the term today. In a 1992 NeurIPS paper, Lorien Pratt described Discriminability-Based Transfer, which initialized a target network from weights learned for a related source task and rescaled those weights according to an information measure. The reported experiments found faster target learning than random initialization in the evaluated networks.^[4] That result was specific to the paper's tasks and algorithm, but it established the source-weight and target-task pattern used by later neural transfer methods.

Deep neural networks made learned representations a central object of transfer. Yosinski and coauthors studied one ImageNet-trained convolutional network and found that transferability depended on both layer position and the distance between the paper's source and target tasks. Their experiments separated specialization of higher-layer features from optimization problems caused by splitting co-adapted layers.^[5] Later analysis showed that successful initialization can reflect more than direct feature reuse. Neyshabur and coauthors found, in their studied image settings, that low-level data statistics and the optimization basin reached from pretrained parameters also contributed to transfer behavior.^[6] These papers do not support a universal rule that every early layer is general or every late layer is task-specific.

In computer vision, DeCAF evaluated activations from a supervised convolutional neural network as generic features on several recognition tasks.^[7] Razavian and coauthors likewise evaluated fixed features from an available pretrained convolutional model with simple downstream classifiers across a set of visual tasks.^[8] Together, these studies helped establish fixed pretrained features as a practical baseline, while later controlled work clarified its limits.

Adaptation strategies

The choice of adaptation strategy determines which quantities are learned on target data.

Scroll sideways for more →

Fixed representations and fine-tuning

With a fixed representation, target inputs pass through a frozen pretrained model and only a new head or downstream learner is trained. This isolates the quality of the learned representation from changes to the backbone. It is also the basis of a linear probe, where the head is linear.

Fine-tuning continues optimization on target data. Full fine-tuning updates all selected parameters, while partial fine-tuning updates only selected blocks or layers. The appropriate comparison depends on the question. A fixed-feature result tests reuse without changing the representation; a fine-tuned result tests the combination of initialization and target optimization. Their training cost, checkpoint storage, and behavior under distribution shift can differ.

Kornblith and coauthors compared 16 ImageNet classification networks on 12 image classification data sets. In that controlled study, ImageNet accuracy was strongly correlated with average transfer accuracy for both fixed features and fine-tuning, but some regularizers that improved ImageNet accuracy produced worse transferable penultimate-layer features. The paper also found minimal pretraining benefit on two fine-grained target data sets.^[9] This supports source-benchmark accuracy as evidence within the study's regime, not as a universal proxy for transfer quality.

Training from scratch remains an essential baseline. He and coauthors reported that standard object-detection and instance-segmentation models trained from random initialization on COCO matched their ImageNet-pretrained counterparts after longer training in the paper's configurations. Pretraining accelerated early convergence but did not necessarily improve final target accuracy there.^[10] A transfer result should therefore distinguish faster optimization from better final generalization.

Pretraining and adaptation in language

Several influential natural language processing systems illustrate different transfer mechanisms. ELMo supplied contextual representations learned by a bidirectional language model to downstream architectures and let each task learn a combination of internal layers.^[11] ULMFiT instead developed a procedure for fine-tuning a pretrained language model for text classification, with techniques intended to control overfitting and forgetting in the paper's experiments.^[12]

BERT pretrained a bidirectional Transformer encoder with masked-token and sentence-level objectives, then fine-tuned the shared parameters with a small task-specific output layer.^[13] T5 compared objectives, architectures, data, and transfer procedures in a unified text-to-text framework rather than treating one pretraining recipe as fixed.^[14] These works demonstrate successful transfer on their reported natural-language benchmarks. They do not show that one representation or objective is best for every language, domain, or downstream task.

Transfer can also span modalities or languages. CLIP learned paired image-text representations from a large collection of image-caption pairs and evaluated zero-shot classification by expressing class concepts in natural language.^[15] XLM-R evaluated multilingual masked-language-model pretraining for cross-lingual transfer and documented tradeoffs involving model capacity, language count, and low-resource languages.^[16] In speech recognition, wav2vec 2.0 pretrained representations from unlabeled speech and then fine-tuned on transcribed audio, including experiments with limited labeled data.^[17] Each result depends on the paper's data collection, benchmark, and evaluation protocol.

Reinforcement learning, simulation, and medicine

Transfer in reinforcement learning can reuse a learned policy, value information, state representation, action mapping, or experience from source tasks. Taylor and Stone's survey emphasized that evaluation can target several different objectives, including faster learning, better asymptotic performance, or the ability to solve a task that otherwise remains out of reach. Those objectives should not be collapsed into one score.^[18]

Simulation-to-real transfer is a domain-shift problem when training observations are generated by a simulator and deployment observations come from physical sensors. Tobin and coauthors randomized camera position, lighting, object position, and textures in simulation, then used the trained detector for a real robotic localization and grasping demonstration without additional real-image training. That paper is evidence for its object-localization setup, not a general guarantee that randomization closes every simulation-to-real gap.^[19]

Medical transfer also illustrates why source-target fit must be tested. Raghu and coauthors evaluated ImageNet transfer on two medical-imaging tasks and reported little performance benefit from the transferred weights in those experiments; lightweight models performed comparably to the larger ImageNet architectures. They found some benefits related to parameterization and weight scaling rather than sophisticated feature reuse.^[20] The result does not establish that natural-image pretraining is ineffective for all medical tasks. It shows why clinical modality, target size, resolution, and task structure must be evaluated directly.

The GPT-3 paper evaluated task examples supplied only through text interaction, without gradient updates or fine-tuning.^[21] This created an important boundary between changing a model for a target task and eliciting target behavior from a fixed model.

Parameter-efficient adaptation

As pretrained models grew, researchers developed methods that train a small task-specific state while freezing most or all backbone parameters.

Houlsby and coauthors inserted adapter modules into BERT and trained the adapters while keeping the original network fixed. On the paper's GLUE experiments, adapters scored 80.0 versus 80.4 for full fine-tuning on the paper's aggregate measure while adding 3.6 percent task-specific parameters per task.^[22] These figures are results for that architecture, adapter design, and benchmark.

Prefix-tuning optimized continuous task-specific vectors that later tokens could attend to while freezing the language model. In Li and Liang's full-data table-to-text experiments, updating 0.1 percent of the parameters produced results comparable to or better than full fine-tuning; in the paper's XSUM summarization experiment, the 0.1-percent setting underperformed full fine-tuning, and a 2-percent setting remained slightly lower on ROUGE-L.^[23] Prompt tuning similarly learned soft input prompts for a frozen T5 model. Lester and coauthors found that its performance approached full model tuning as the evaluated model scale increased.^[24]

LoRA represents a weight update with trainable low-rank matrices while freezing the original weights. Hu and coauthors reported large reductions in trainable parameters and optimizer memory in their evaluated language models, with updates that could be merged into the original weights for inference.^[25] QLoRA combined low-rank adapters with a frozen 4-bit quantized backbone. Dettmers and coauthors reported fine-tuning a 65-billion-parameter model on a single 48 GB GPU in their implementation while comparing the result with a 16-bit fine-tuning baseline.^[26] These are method-specific measurements. Actual memory, throughput, and output quality depend on model architecture, sequence length, batch size, optimizer, quantization implementation, and hardware.

Parameter-efficient methods reduce the target-trained state, but they do not eliminate the cost of running or storing the shared backbone. They can also impose method-specific limits, such as added modules, prompt length, low-rank capacity, or quantization error. A fair comparison reports trainable parameters, peak memory, training time, checkpoint size, inference latency, and target quality under a matched evaluation protocol.

In-context learning and the boundary of adaptation

In-context learning uses instructions or examples in a model's input without updating its parameters. The GPT-3 paper evaluated zero-shot, one-shot, and few-shot prompting with no gradient updates or fine-tuning.^[21] It called these settings zero-shot and few-shot transfer, but the mechanism differs from parameter adaptation: the model state remains fixed and the examples exist only in the context window.

This distinction matters when comparing data and compute. A prompted model consumes demonstration tokens at inference time, while a fine-tuned or adapter-based model spends training compute and stores task-specific state. Few-shot learning is a broader data regime and can use either kind of mechanism. Reports should state whether examples changed model parameters, selected a model or prompt, or were supplied only at inference.

Evaluation

Transfer should be evaluated against alternatives that answer different questions:

1. Training from scratch measures what the target data and architecture can achieve without source initialization.
2. A frozen representation or linear probe measures how useful the pretrained representation is without backbone adaptation.
3. Partial and full fine-tuning measure the benefit and cost of updating progressively more of the backbone.
4. A parameter-efficient method measures whether a compact target state achieves an acceptable tradeoff.
5. Source-only evaluation, where meaningful, reveals how much performance is present before target adaptation.

A credible experiment holds the architecture, target split, metric implementation, and selection rule as constant as possible. It reports training schedules and compute because a pretrained model can converge sooner while a randomly initialized model reaches a similar endpoint after more optimization.^[10] Hyperparameters should be tuned without using the test set, and comparisons should include multiple random seeds when training variation is material.

Target evaluation must match deployment. An in-distribution test set estimates performance near the target training distribution. Additional test sets are needed when deployment includes new sites, devices, languages, time periods, demographic groups, visual styles, or other shifts. A single aggregate benchmark can conceal class- or group-specific failures.

Pretraining data also require documentation. Overlap between pretraining and target test examples can inflate an apparent transfer result. The GPT-3 authors identified methodological issues for some data sets when evaluating a model trained on large web corpora.^[21] For large reusable models, documenting source data may be difficult, but uncertainty about overlap should be reported rather than treated as absence of leakage.

The relevant outcome is not always final accuracy. Depending on the purpose, an evaluation may measure labeled-example efficiency, time to a target score, final score at matched compute, calibration, robustness, memory, energy, checkpoint storage, or inference cost. No single number captures all of these.

Negative transfer and other failure modes

Negative transfer needs an explicit comparator. Wang and coauthors defined it relative to the target performance of a specified non-transfer algorithm and showed that it is algorithm-dependent.^[3] A source can therefore help one adaptation method and harm another. Describing two domains as semantically related is not enough to predict the outcome.

For domain adaptation in binary classification, Ben-David and coauthors derived target-error bounds involving source error, a hypothesis-dependent divergence between source and target distributions, and the error of the best shared hypothesis. The assumptions matter: small measurable domain divergence does not by itself guarantee a good target predictor if no hypothesis performs well on both domains.^[27] Domain-adversarial neural networks implement one response to distribution shift by learning features that support the source task while making source and target domains difficult to distinguish. Ganin and coauthors evaluated that approach on selected sentiment, image-classification, and person-reidentification problems.^[28] Neither result establishes that invariant features preserve every target-relevant distinction.

Full fine-tuning can also damage a useful representation. Kumar and coauthors compared linear probing and full fine-tuning on ten distribution-shift data sets. In their experiments, fine-tuning averaged higher in-distribution accuracy but lower out-of-distribution accuracy than linear probing when the pretrained features were good and the shift was large. A two-stage linear-probe-then-fine-tune procedure improved the reported tradeoff.^[29] This is evidence for the tested models and shifts, not a fixed ranking of adaptation strategies.

Other failure modes include:

• Source mismatch: source features or labels do not support the target decision and may encourage spurious shortcuts.
• Representation distortion: target optimization changes features that were useful outside the target training distribution.
• Overfitting: a high-capacity adaptation procedure fits a small target sample without improving held-out target behavior.
• Pretraining contamination: target test data or near-duplicates appear in the source corpus.
• Inherited defects: biases, security weaknesses, factual errors, or other limitations in a shared model propagate to adapted systems.
• Misleading efficiency claims: a method reports trainable parameters while omitting backbone memory, optimizer state, preprocessing, or inference cost.

The foundation-model report by Bommasani and coauthors described this last category of inherited risk as a consequence of homogenization: one broadly reused model provides leverage, but its defects can affect many downstream systems.^[30] Adaptation does not by itself remove those defects.

Relationship to adjacent methods

Transfer learning overlaps with several fields but should not erase their separate scope.

Scroll sideways for more →

Clear reports identify the source, target, transferred object, adaptation rule, comparator, and evaluation distribution. Without those details, a statement that a system "uses transfer learning" says little about what was transferred or whether the transfer helped.

References

1. ^Sinno Jialin Pan and Qiang Yang, "A Survey on Transfer Learning," IEEE Transactions on Knowledge and Data Engineering 22(10), 2010. doi.org/...TKDE.2009.191
2. ^Fuzhen Zhuang et al., "A Comprehensive Survey on Transfer Learning," Proceedings of the IEEE 109(1), 2021. doi.org/...JPROC.2020.3004555
3. ^Zirui Wang, Zihang Dai, Barnabas Poczos, Jaime Carbonell, and Bryan Catanzaro, "Characterizing and Avoiding Negative Transfer," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. openaccess.thecvf.com/..._Transfer_CVPR_2019_paper
4. ^Lorien Y. Pratt, "Discriminability-Based Transfer between Neural Networks," Advances in Neural Information Processing Systems 5, 1992. proceedings.neurips.cc/...e83c559be18d40c-Abstract
5. ^Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson, "How transferable are features in deep neural networks?," Advances in Neural Information Processing Systems 27, 2014. proceedings.neurips.cc/...c93f8580abbb330-Abstract
6. ^Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang, "What is being transferred in transfer learning?," Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...a4f3e7a127b44e0-Abstract
7. ^Jeff Donahue et al., "DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition," Proceedings of Machine Learning Research 32, 2014. proceedings.mlr.press/...donahue14
8. ^Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson, "CNN Features Off-the-Shelf: An Astounding Baseline for Recognition," CVPR Workshops, 2014. openaccess.thecvf.com/...the-Shelf_2014_CVPR_paper
9. ^Simon Kornblith, Jonathon Shlens, and Quoc V. Le, "Do Better ImageNet Models Transfer Better?," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. openaccess.thecvf.com/...er_Better_CVPR_2019_paper
10. ^Kaiming He, Ross Girshick, and Piotr Dollar, "Rethinking ImageNet Pre-training," Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. openaccess.thecvf.com/...-Training_ICCV_2019_paper
11. ^Matthew E. Peters et al., "Deep Contextualized Word Representations," Proceedings of NAACL-HLT, 2018. aclanthology.org/N18-1202
12. ^Jeremy Howard and Sebastian Ruder, "Universal Language Model Fine-tuning for Text Classification," Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018. aclanthology.org/P18-1031
13. ^Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," Proceedings of NAACL-HLT, 2019. aclanthology.org/N19-1423
14. ^Colin Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research 21(140), 2020. jmlr.org/...20-074
15. ^Alec Radford et al., "Learning Transferable Visual Models From Natural Language Supervision," Proceedings of Machine Learning Research 139, 2021. proceedings.mlr.press/...radford21a
16. ^Alexis Conneau et al., "Unsupervised Cross-lingual Representation Learning at Scale," Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. aclanthology.org/2020.acl-main.747
17. ^Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations," Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...3227870bb6d7f07-Abstract
18. ^Matthew E. Taylor and Peter Stone, "Transfer Learning for Reinforcement Learning Domains: A Survey," Journal of Machine Learning Research 10, 2009. jmlr.org/...taylor09a
19. ^Josh Tobin et al., "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World," IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017. arxiv.org/...1703.06907
20. ^Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio, "Transfusion: Understanding Transfer Learning for Medical Imaging," Advances in Neural Information Processing Systems 32, 2019. proceedings.neurips.cc/...6a4ac4a1e378b91-Abstract
21. ^Tom B. Brown et al., "Language Models are Few-Shot Learners," Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...18bfb8ac142f64a-Abstract
22. ^Neil Houlsby et al., "Parameter-Efficient Transfer Learning for NLP," Proceedings of Machine Learning Research 97, 2019. proceedings.mlr.press/...houlsby19a
23. ^Xiang Lisa Li and Percy Liang, "Prefix-Tuning: Optimizing Continuous Prompts for Generation," Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021. aclanthology.org/2021.acl-long.353
24. ^Brian Lester, Rami Al-Rfou, and Noah Constant, "The Power of Scale for Parameter-Efficient Prompt Tuning," Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. aclanthology.org/2021.emnlp-main.243
25. ^Edward J. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," International Conference on Learning Representations, 2022. arxiv.org/...2106.09685
26. ^Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs," Advances in Neural Information Processing Systems 36, 2023. proceedings.neurips.cc/...049b-Abstract-Conference
27. ^Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan, "A Theory of Learning from Different Domains," Machine Learning 79, 2010. doi.org/...s10994-009-5152-4
28. ^Yaroslav Ganin et al., "Domain-Adversarial Training of Neural Networks," Journal of Machine Learning Research 17(59), 2016. jmlr.org/...15-239
29. ^Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang, "Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution," International Conference on Learning Representations, 2022. arxiv.org/...2202.10054
30. ^Rishi Bommasani et al., "On the Opportunities and Risks of Foundation Models," arXiv:2108.07258, 2021. arxiv.org/...2108.07258