Supervised fine-tuning

Supervised fine-tuning (SFT) is supervised training applied to a model that has already been trained. The new training set supplies target outputs for specified inputs, and optimization changes some or all of the model's parameters so that its predictions better match those targets. SFT is therefore a form of transfer learning, not a separate model architecture.^[1][2] It is used across machine learning, including image classification, speech recognition, and natural language processing. In work on large language models (LLMs), the term usually refers to training a pretrained model on prompt-response demonstrations or other instruction-formatted examples.^[7][10]

SFT can specialize a model for one task, adapt it to a domain, or teach a general model to respond to many kinds of natural-language instructions. The last use is commonly called instruction tuning.^[7][8][9] These descriptions concern the data and objective. They do not specify which parameters are trainable: both full-parameter training and parameter-efficient fine-tuning can use the same supervised objective.^[18][20]

In many assistant-training pipelines, SFT is the first post-pretraining stage. Later stages may optimize preferences through reinforcement learning from human feedback, Direct Preference Optimization (DPO), rejection sampling, or reinforcement learning with verifiable rewards.^[10][17][25] Results from a complete pipeline cannot be attributed to SFT alone unless the study reports an SFT-only ablation. For example, the widely cited finding that a 1.3-billion-parameter InstructGPT model was preferred to the 175-billion-parameter GPT-3 baseline concerns the final InstructGPT policy trained with SFT and subsequent reinforcement learning, not the SFT checkpoint by itself.^[10]

Scope and terminology

Fine-tuning is a broad term for additional training after an initial model has been fitted. SFT narrows that term to additional training with examples whose desired outputs are available. Several neighboring methods also continue training, but differ in their data, objective, or source of supervision.

Scroll sideways for more →

"Post-training" is an umbrella term. It can include SFT, preference optimization, safety tuning, tool-use training, model merging, or reinforcement learning. A post-trained model is not necessarily an SFT-only model.^[17][36]

Historical development

Transfer learning and task fine-tuning

Transfer learning predates modern language models. Its central idea is to reuse representations learned for a source setting when fitting a related target setting. Pan and Yang's 2009 survey organized transfer-learning methods by what is transferred and by the relation between source and target domains and tasks.^[1] Later experiments on convolutional networks showed that learned features can remain useful after transfer, while also showing that transferability depends on layer depth and on how different the source and target tasks are.^[2] These findings do not imply that early layers are always general or that later layers should always be retrained. Layer behavior depends on the architecture, data, and target task.

In language modeling, ULMFiT combined general-domain language-model pretraining, optional target-domain language-model adaptation, and supervised classifier fine-tuning. It also introduced optimization choices intended to avoid damaging useful pretrained representations.^[3] BERT then used masked-language-model pretraining followed by task-specific fine-tuning, adding a small output layer for each downstream task while updating BERT's parameters.^[4] T5 expressed many text tasks in a text-to-text format, which made it possible to train one architecture on mixtures of tasks with textual inputs and outputs.^[5]

These systems established the pretrain-then-fine-tune pattern, but the label "SFT" was not yet restricted to assistant demonstrations. A BERT classifier trained on labeled entailment examples and a decoder-only LLM trained on instruction-response conversations both use supervised fine-tuning, although their output structures and evaluation methods differ.

Instruction tuning

Instruction tuning extended task fine-tuning from one dataset to mixtures in which tasks were described with natural-language instructions. FLAN instruction-tuned a 137-billion-parameter model on more than 60 tasks and evaluated zero-shot transfer to held-out task clusters. Under that paper's templates and datasets, zero-shot FLAN exceeded the reported zero-shot GPT-3 result on 20 of 25 evaluated datasets.^[7] Natural Instructions and T0 likewise studied cross-task generalization with explicit task descriptions or prompted task mixtures.^[8][9]

The scale and composition of instruction mixtures then expanded. The Flan-PaLM study combined 1,836 tasks and included examples with and without chain-of-thought annotations. Its normalized average across six reported evaluation settings improved by 9.4 percentage points for the 540-billion-parameter model relative to the corresponding PaLM model, under the study's few-shot and chain-of-thought protocols.^[12] The Flan Collection report documented how task balancing, templates, zero-shot and few-shot formats, and chain-of-thought data affected that training mixture.^[13] These results are evidence about those experiments, not a universal gain expected from instruction tuning.

SFT in assistant post-training

InstructGPT made "supervised fine-tuning" a standard name for the demonstration stage of an LLM alignment pipeline. OpenAI fine-tuned GPT-3 models on about 13,000 training prompts with labeler-written demonstrations, then trained a reward model on comparisons and optimized a policy with proximal policy optimization.^[10] The SFT models were trained for 16 epochs in that experiment, with learning rates and batch sizes selected for the studied model sizes. Those values were protocol choices, not general defaults.^[10]

Later open recipes used different data and schedules. Llama 2 began with public instruction data, then collected 27,540 vendor-produced SFT annotations. Its reported SFT run masked loss on user-prompt tokens, optimized answer tokens for two epochs, and used a sequence length of 4,096 tokens.^[37] Tulu 2 and Tulu 3 published broader mixtures and evaluation recipes; Tulu 3 separated SFT, DPO, and reinforcement learning with verifiable rewards and released decontamination tools with its data and code.^[16][17] The open OLMo project likewise reported base, SFT, and SFT-plus-DPO variants, which permits comparisons among stages within one model family.^[42]

Training objective

Supervised examples

Let a training set contain examples (x_i, y_i), where x_i is an input and y_i is the desired output. For a classifier, y_i may be a class label. For a sequence model, it may be a target sequence. In instruction tuning, x_i can contain a system instruction, conversation history, tools, retrieved context, and the current user turn, while y_i contains the assistant response that the model should imitate.

For an autoregressive language model, a common SFT objective is conditional negative log-likelihood:

L_SFT(theta) = - sum_i sum_t m_i,t log p_theta(y_i,t | x_i, y_i,<t)

Here, theta denotes trainable parameters, y_i,t is the target token at position t, and m_i,t is a mask that decides whether that token contributes to the loss. The model is trained with teacher forcing: at training time, the prefix contains the reference tokens that precede the current target token. Backpropagation computes gradients of the loss, and an optimizer updates the trainable parameters.^[10][37]

The mask is part of the method and must be reported. Some runs apply loss only to assistant or completion tokens. Llama 2 did this by zeroing loss on user-prompt tokens.^[37] Other task formulations train on every token in a serialized example. Multi-turn datasets may supervise one assistant turn, all assistant turns, or selected turns. These choices change the objective even when all are called SFT.

For classification, regression, segmentation, or other supervised tasks, the output head and loss follow the task. Cross-entropy is common for discrete classes and next-token prediction, but SFT does not require one universal loss function. The phrase describes when and with what supervision the model is trained, not a single equation for every model type.

Full and partial parameter updates

In full-parameter SFT, gradients update every trainable weight in the model. This gives the optimizer access to the model's full parameter space, but also requires optimizer states and gradients for the full model and normally produces a complete new checkpoint.

Partial fine-tuning freezes some pretrained parameters. A run may update only an output head, selected layers, biases, soft prompts, adapters, or low-rank parameterizations. LoRA represents weight updates with trainable low-rank matrices while keeping the original weight matrix frozen. In the original GPT-3 175B experiment, the authors reported a 10,000-fold reduction in trainable parameters and a threefold reduction in GPU memory relative to their full fine-tuning setup, while obtaining comparable or better results on the tasks they tested.^[18] Those ratios depend on the chosen rank, target modules, optimizer, and baseline.

QLoRA keeps a quantized base model frozen and backpropagates through it into LoRA adapters. Its paper reported fine-tuning a 65-billion-parameter model on one 48 GB GPU and evaluated the resulting Guanaco models with automatic and human comparisons.^[19] QLoRA is therefore both quantized training and parameter-efficient SFT. It is not evidence that every 65-billion-parameter model, sequence length, or dataset fits the same device.

PEFT can trade adaptation capacity for storage and memory savings. In controlled programming and mathematics experiments, Biderman and colleagues found that conventional low-rank LoRA settings learned less of the target distribution than full fine-tuning, while preserving more performance outside the target domain. They also found that the full fine-tuning updates had much higher effective rank than the tested LoRA configurations.^[21] This does not establish that LoRA is always inferior or always safer; the result was conditional on the models, ranks, datasets, token budgets, and evaluations in that study.

Data design

What counts as a label

The target in SFT need not be a short categorical label. It can be a translation, summary, program, tool call, structured record, answer with a rationale, safe refusal, or an entire assistant turn. A demonstration may be written by a human, transformed from an existing labeled dataset, generated by another model and filtered, produced by rejection sampling, or distilled from a teacher. The provenance matters because it affects error patterns, licensing, privacy, and what the student is able to learn.

Instruction examples commonly include several fields:

Scroll sideways for more →

The serialization of these fields is not cosmetic. Chat templates add role tokens, separators, beginning and end markers, and sometimes generation prompts. A model trained with one serialization can behave differently when served with another. Tokenization also determines the length of each example and where truncation occurs.

Human demonstrations

Human-written targets can encode domain expertise, interaction norms, safety policies, and formatting that are difficult to obtain from raw text. They can also contain disagreement and annotation artifacts. InstructGPT screened and trained labelers, supplied detailed instructions, and used separate held-out labelers for part of its evaluation.^[10] Llama 2 reported that annotation vendor and platform choices produced noticeably different downstream results and stopped SFT annotation after 27,540 examples when model-generated candidates became competitive with its handwritten data under internal review.^[37]

Annotation quality cannot be inferred from whether a response was written by a person. A useful process defines the task, trains annotators, measures agreement where applicable, reviews edge cases, records revisions, and separates authors of training targets from evaluators when feasible. High-stakes domains also require subject-matter review and application-specific validation; SFT does not make a model a qualified clinician, lawyer, or financial adviser.

Synthetic demonstrations

Synthetic data can broaden task coverage and reduce direct annotation. Self-Instruct began with 175 human-written seed tasks, prompted GPT-3 to generate new instructions and instances, filtered invalid or similar generations, and produced 52,445 instructions with more than 82,000 instances. The paper fine-tuned GPT-3 on that generated data and evaluated it on held-out author-written tasks.^[11] The method demonstrates one way to generate SFT data; it also illustrates why generation, filtering, and evaluation must be treated as separate stages.

Synthetic targets inherit the generator's errors, biases, style, and blind spots. Filtering only for fluent formatting can preserve factual errors. Filtering against a reward model can select responses that exploit that model's preferences. When a stronger model supplies targets to a weaker one, the process may overlap with knowledge distillation, but a dataset's being synthetic does not by itself establish a controlled teacher-student distillation setup.^[28]

Public conversational datasets illustrate another collection model. OpenAssistant Conversations used a crowdsourced tree interface for prompts, replies, rankings, and moderation. Its first release contained 161,443 messages across 66,497 conversation trees in 35 languages, with 461,292 quality ratings; 8,576 messages were synthetic and the remainder were human-submitted.^[15] These figures describe the released corpus. They do not certify every message as correct or safe, and the paper documents moderation limits and uneven language representation.^[15]

Modern pipelines often combine human prompts with model-generated candidates. Llama 3 sampled multiple outputs per prompt and used a reward model to select a candidate, with the paper reporting typical candidate counts between 10 and 30 for that rejection-sampling stage.^[36] DeepSeek-R1 used several distinct SFT stages: thousands of cold-start reasoning examples before reinforcement learning, followed by about 600,000 filtered reasoning examples and about 200,000 non-reasoning examples. The released table lists 804,745 second-stage supervised examples across math, code, STEM, logic, and general domains.^[35] These counts describe those specific recipes, not a recommended SFT dataset size.

Quality, quantity, and diversity

There is no model-independent minimum or optimum number of SFT examples. Required data depends on the base model, task complexity, target distribution, label noise, output diversity, and evaluation threshold. Claims that a fixed count such as 10, 50, or 1,000 examples is generally sufficient omit these variables.

LIMA is evidence for a narrower proposition. The study fine-tuned LLaMA 65B on exactly 1,000 curated prompt-response sequences, about 750,000 tokens, and compared it with contemporary systems in its evaluation setup. The authors used the result to propose the "Superficial Alignment Hypothesis," under which pretraining supplies most knowledge and capability while alignment selects response formats and interaction styles.^[14] It remains a hypothesis supported by the paper's experiments, not a theorem about every task or base model.

Llama 2 independently reported that replacing millions of third-party examples with tens of thousands of higher-quality annotations improved its internal results.^[37] By contrast, Tulu 3 found continued average improvement when increasing stratified samples up to its full 939,344-example SFT mixture, while TruthfulQA performance fell as more mixture data was added.^[17] Together these studies show that quality and mixture composition matter, but they do not support the slogan that less data is always better.

Data selection itself is an experimental variable. The DEITA study measured candidate examples along complexity, quality, and diversity dimensions, then evaluated a selection method using 6,000 SFT samples and a later DPO stage.^[32] CLEAR used model confidence to identify examples for filtering or correction and tested the resulting curation pipeline across several datasets and models.^[41] Both are proposed methods with reported experiments. Neither makes automatic scoring a substitute for source provenance, expert review, or held-out evaluation.

Mixtures and sampling

A multi-skill dataset needs an explicit sampling policy. Concatenating datasets in their original sizes may cause one high-volume source to dominate. Alternatives include uniform task sampling, temperature-based sampling, caps, upsampling of scarce skills, curriculum schedules, or manually chosen weights. Each changes the effective training distribution.

Tulu 3 documents this problem directly. Its authors built skill-specific mixtures, tested additions and removals, decontaminated against the evaluation suite, and reported SFT-only checkpoints before DPO and reinforcement learning.^[17] The 2025 controlled study by Harada and colleagues trained more than 1,000 SFT models across base models and task datasets. It found some relationships that persisted across models and others that varied, with training-data perplexity serving as a useful predictor in that experimental grid.^[34] The finding argues for model-specific validation rather than a universal mixture recipe.

Splits, duplicates, and contamination

Training, validation, and test sets should represent the intended use while remaining independent enough to measure generalization. Duplicate or near-duplicate prompts can cross splits through paraphrases, templating, synthetic generation, or aggregation of public datasets. Benchmark questions can also enter training mixtures, inflating scores without improving the underlying capability.

Decontamination requires a declared rule and threshold. Exact string matching catches literal overlap but misses paraphrases; embedding or model-based similarity can create false positives and false negatives. Tulu 3 released its decontamination implementation and separated development evaluations from unseen evaluations.^[17] An honest report names the detector, normalized fields, threshold, removed counts, and whether reference answers, rationales, or only prompts were compared.

End-to-end workflow

Define the target behavior

The first decision is whether training is needed. Prompting or retrieval may be preferable when the desired change is temporary, knowledge changes frequently, or a fixed model already performs the task with reliable context. SFT is appropriate when repeated behavior, a stable output schema, a task mapping, or interaction policy should be learned into parameters.

The target should be testable. "Make the model better" is not a sufficient specification. A project can instead define the supported languages, prompt distribution, acceptable output formats, refusal policy, latency limits, and task metrics. It should also state which capabilities must not regress.

Choose and audit a base model

Base-model selection constrains what SFT can achieve. Relevant factors include architecture, tokenizer, context length, pretraining data and cutoff, license, parameter access, quantization support, and baseline performance. A supervised dataset can teach a response convention or specialize an existing capability, but it does not guarantee reliable acquisition of knowledge or reasoning absent from the base model.

The exact starting checkpoint matters. A base model, an instruction-tuned model, and a chat model from the same family have different prior behavior. Fine-tuning an already safety-aligned assistant may overwrite some alignment, while fine-tuning a raw base model may require demonstrations that establish basic interaction structure.^[29][37]

Prepare examples

A reproducible preparation pipeline records source identifiers and licenses, removes disallowed personal or confidential data, normalizes encodings, applies the exact chat template, tokenizes with the target tokenizer, and records truncation. It also checks class or skill balance, duplicate rates, language distribution, output length, refusals, tool schemas, and invalid examples.

Packing multiple short examples into one sequence can reduce padding, but boundaries must remain explicit. Loss masks must not leak one example's target into another example's prompt. For multi-turn chat, the pipeline should state whether earlier assistant turns contribute to loss. For long examples, truncating from the left, right, or middle can change the task.

Configure optimization

SFT usually starts from pretrained weights and applies mini-batch gradient descent through an optimizer such as AdamW. Learning rate, effective batch size, sequence length, number of epochs or tokens, warmup, weight decay, dropout, gradient clipping, precision, and checkpoint selection interact. Values published for one model are starting evidence only for that setting.

The contrast among published recipes makes this clear. InstructGPT selected different learning rates and batch sizes for its 1.3B, 6B, and 175B models and trained for 16 epochs.^[10] Llama 2 reported an initial learning rate of 2 x 10^-5, batch size 64, 4,096-token sequences, and two epochs for its SFT run.^[37] Tulu 3 reported two epochs but used 5 x 10^-6 for its 8B model and 2 x 10^-6 for its 70B model after hyperparameter searches, with effective batch size 128.^[17] Copying one row into a universal "best practices" table would be misleading.

Validation should monitor more than training loss. A checkpoint with the lowest imitation loss may not have the best task accuracy, instruction adherence, calibration, human preference, or safety. InstructGPT reported that its SFT validation loss began to overfit after one epoch, while later epochs still improved reward-model scores and human preference, so it selected the SFT checkpoint using reward-model validation score.^[10] That selection procedure was particular to a pipeline with a trained reward model.

Evaluate before and after training

Evaluation begins with a frozen baseline and unchanged test protocol. For a classifier, appropriate measures may include accuracy, precision, recall, F1, calibration, and performance by subgroup. For generation, evaluation can include exact match, execution tests, factuality checks, constraint satisfaction, reference-based metrics, and blinded human judgments. The metric must match the intended behavior.

General assistant evaluation needs several axes because one aggregate score can hide regressions. Useful categories include instruction following, factuality, reasoning, coding, multilingual performance, long-context behavior, safety, and out-of-distribution tasks. Tulu 3 used a multi-task suite with development and unseen sets, while OLMo reported base, SFT, and SFT-plus-DPO checkpoints on the same evaluation family.^[17][42]

Instruction-following benchmarks also have limits. IFEval uses verifiable constraints, which reduces reliance on subjective judging but covers a defined set of instruction types.^[40] MT-Bench uses multi-turn questions and an LLM judge; its paper analyzed agreement, position bias, verbosity, and limits of model judging.^[39] Length-Controlled AlpacaEval was introduced because ordinary pairwise automatic evaluation can favor longer answers; its method reports a length-controlled win rate under a specific reference and judge.^[38] Benchmark name, version, prompt template, decoding settings, judge, reference model, and confidence intervals should accompany any score.

Deploy and monitor

Passing an offline test does not establish production reliability. Deployment changes prompt distributions, tools, retrieval context, user languages, and incentives. Monitoring can track schema failures, abstentions, safety incidents, task errors, drift, and subgroup performance while respecting privacy. Rollback requires preserving the base checkpoint, adapter or fine-tuned weights, tokenizer, template, data manifest, code version, and evaluation report.

Relationship to other post-training methods

Continued pretraining

Continued pretraining applies a self-supervised language-model objective to domain or task text. The "Don't Stop Pretraining" study distinguished domain-adaptive pretraining and task-adaptive pretraining, then evaluated them before supervised task fine-tuning.^[6] Continued pretraining can adapt vocabulary and distributional knowledge without labeled responses. SFT instead optimizes supplied target outputs. A pipeline may use both in sequence, and calling the first stage "unsupervised SFT" obscures this distinction.

Preference optimization and reinforcement learning

SFT imitates demonstrated outputs. Preference methods learn from relative judgments or rewards and can optimize sequence-level properties for which there is no single reference response. Christiano and colleagues trained reward predictors from human comparisons and used those learned rewards for reinforcement learning, establishing a general preference-learning approach before current LLM pipelines.^[24]

InstructGPT used SFT to initialize a policy, trained a reward model from ranked model outputs, and then applied proximal policy optimization.^[10] DPO derived a classification-style objective that directly uses preferred and rejected response pairs relative to a reference policy, without separately fitting and sampling from a reward model during training.^[25] DPO is not SFT even though both can be implemented with gradient-based language-model training.

Constitutional AI combined a supervised phase, in which a model generated critiques and revisions under written principles, with a reinforcement-learning-from-AI-feedback phase.^[26] DeepSeek-R1 reported a cold-start SFT stage, reinforcement learning, rejection-sampled SFT, and a further reinforcement-learning stage.^[35] Tulu 3 likewise reported SFT, DPO, and reinforcement learning with verifiable rewards as distinct checkpoints.^[17] These examples show why "post-training" results must be decomposed by stage.

SFT and reinforcement learning for reasoning

No experiment establishes a universal ordering between SFT and reinforcement learning. The 2025 "SFT Memorizes, RL Generalizes" paper compared the methods on a card-game rule-variation task and a visual navigation environment. In those settings, outcome-reward reinforcement learning generalized better to unseen variants, while SFT helped stabilize output format before reinforcement learning.^[33] This finding is limited to the paper's models, environments, data, and reward design.

SRFT proposed a single-stage objective combining demonstration loss with reinforcement learning and reported gains on its selected mathematical reasoning benchmarks and out-of-distribution sets.^[43] A 2026 review organized work on SFT, reinforcement learning, sequential pipelines, and hybrid objectives, and emphasized that data availability, reward verifiability, task structure, model capacity, and distribution shift affect the choice.^[44] Hybrid methods should therefore be described by their exact loss and sampling procedure rather than treated as a new meaning of SFT.

Retrieval, prompting, and distillation

Retrieval-Augmented Generation retrieves passages and conditions generation on them, allowing source material to change without updating model weights.^[27] SFT can teach a model how to use retrieved evidence, cite passages, or follow a response schema, but it is not a reliable substitute for retrieval when facts change frequently.

Prompt Engineering changes the instructions or examples supplied at inference time. It is cheaper to revise and does not create a new checkpoint, but consumes context and must be repeated for each request. SFT can internalize a stable behavior, although the learned behavior may still depend on using the training-compatible chat template.

Distillation trains a student from a teacher's probability distribution, logits, internal representations, or generated outputs. Hinton, Vinyals, and Dean described soft targets that convey information beyond a hard class label.^[28] Modern reasoning pipelines sometimes fine-tune smaller models on trajectories produced by a larger model, which is supervised training and can also be described as distillation when the teacher-student relation is explicit.^[35]

Capabilities and applications

SFT is used for classification, extraction, translation, summarization, question answering, code generation, tool calling, structured output, multimodal tasks, domain adaptation, and conversational behavior. The design differs by application.

Scroll sideways for more →

Instruction tuning can support cross-task transfer when the mixture contains enough variation in tasks and templates.^[7][9][12] It can also teach superficial conventions without producing reliable new competence. Whether SFT elicits an existing capability, teaches a new mapping, or memorizes examples is an empirical question for the chosen base model and evaluation distribution.^[14][33]

Failure modes and risks

Overfitting and memorization

Small supervised datasets paired with high-capacity models can produce low training loss without generalization. Repeated epochs, duplicate examples, narrow templates, and label noise increase this risk. Overfitting can appear as memorized answers, brittle formatting, or loss of performance on paraphrased and out-of-distribution prompts.

Mitigations include independent validation and test sets, deduplication, augmentation that preserves labels, early stopping when justified by the target metric, regularization, and collecting more representative examples. None is automatic. The best checkpoint can differ by metric, and a lower validation loss does not guarantee better human preference or safer behavior.^[10]

Catastrophic forgetting and regressions

Additional training can reduce performance on tasks that were not represented in the SFT mixture. Elastic Weight Consolidation was proposed to slow changes to parameters judged important for earlier tasks, illustrating a regularization approach to catastrophic forgetting.^[22] Rehearsal, replay data, smaller updates, model merging, and PEFT are other investigated strategies.

For LLMs, forgetting depends on update count, trainable capacity, data distribution, and metric. A 2024 scaling study found a trade-off between target-task learning and forgetting in its LoRA experiments and reported shifted power-law relationships with trainable parameters and update steps.^[23] Biderman and colleagues found less out-of-domain forgetting with LoRA than with full fine-tuning in their programming and mathematics experiments, alongside weaker target learning for conventional low-rank settings.^[21] These studies do not show that PEFT prevents forgetting.

A regression suite should compare the base and fine-tuned checkpoints on target tasks, general capabilities, safety policies, multilingual behavior, calibration, and long-context behavior. Results should be reported even when target-task performance improves.

Safety erosion

Fine-tuning an aligned model can change refusal and safety behavior. Qi and colleagues showed that ten adversarial examples were enough to substantially compromise safeguards in their GPT-3.5 Turbo fine-tuning experiment, and that benign fine-tuning datasets could also degrade safety to a lesser degree in the tested models.^[29] The exact attack cost and effectiveness depended on the then-current service and evaluation; the broader result is that alignment is not invariant under further training.

Safety data mixed into SFT can reduce some regressions, but it may compete with other objectives and cannot cover every harmful behavior. Llama 2 included adversarial prompts and safe demonstrations in supervised safety fine-tuning before safety-specific preference training.^[37] Tulu 3 treated safety as a measured skill in its SFT mixture and reported ablations.^[17] A release still needs red teaming and application-level controls after training.

Poisoning and backdoors

Aggregated and user-contributed instruction data create a data poisoning surface. Wan and colleagues inserted as few as 100 poison examples in their instruction-tuning experiments and induced trigger-linked failures across held-out tasks; filtering and reducing model capacity provided only partial protection in their setup.^[30] The result supports provenance controls and targeted testing, not the claim that every public dataset is poisoned.

Backdoors can also persist through later safety training. The Sleeper Agents study constructed proof-of-concept models with conditional harmful behavior and found that supervised fine-tuning, reinforcement learning, and adversarial training did not reliably remove the implanted behavior in its experiments.^[31] The authors did not estimate how likely such deceptive policies are to arise in deployed systems. Their work shows a limitation of behavioral training as a guaranteed removal method.

Practical controls include immutable source manifests, cryptographic hashes, access control over training data, duplicate and anomaly analysis, trigger-oriented tests, manual review of high-influence examples, and retention of a clean rollback checkpoint. Automated filtering is one layer, not proof of safety.

Bias, coverage, and labeler disagreement

SFT learns the distribution of its targets. If groups, dialects, languages, or use cases are absent or labeled inconsistently, the model may reproduce those gaps. A single canonical response can also suppress legitimate variation. Evaluation should examine relevant subgroups and include uncertainty or abstention where the application permits it.

Human demonstrations encode judgments about helpfulness, tone, risk, and correctness. Those judgments vary. Documentation should identify who supplied labels, their instructions and qualifications, how disagreement was handled, and whether affected communities or domain experts participated. A model's agreement with annotators is not automatically agreement with all users.

Evaluation leakage and judge bias

Public benchmarks can enter training data directly or through synthetic generators. Decontamination reduces known overlap but cannot prove absence. Repeated tuning against a public leaderboard can also overfit decisions to that benchmark without literal example leakage.

LLM judges introduce their own preferences. MT-Bench and AlpacaEval made scalable open-ended comparison possible, but their papers documented position, style, and length effects.^[38][39] Reliable evaluation combines deterministic tests where available, multiple benchmarks, blinded human review, and deployment-relevant examples. Scores from different prompts, judges, or benchmark versions are not directly interchangeable.

Reproducibility and reporting

An SFT report is interpretable only when it binds results to the training and evaluation protocol. At minimum, it should record:

Scroll sideways for more →

Open recipes show why this detail matters. OLMo released pretraining and adaptation artifacts, while Tulu 3 released data, checkpoints, code, and decontamination tools for distinct SFT, DPO, and reinforcement-learning stages.^[17][42] Reproducibility still does not mean every claimed result transfers to another base model or use case. It allows the claim to be inspected and tested under a defined protocol.

Research directions

Current work studies data selection, mixture optimization, forgetting, synthetic supervision, safety preservation, and combinations of imitation with exploration. Controlled SFT grids have found model-specific interactions among base checkpoints and training tasks.^[34] Reasoning studies have compared SFT with outcome-reward reinforcement learning and hybrid objectives, with conclusions that depend on the task, data, and reward.^[33][43][44]

Another question is what SFT changes internally. The 2025 study by Harada and colleagues compared layer-wise weight changes across more than 1,000 trained models and reported that mid-layer changes correlated most strongly with gains in its setup.^[34] Such correlational evidence does not supply a universal rule for freezing layers. Architecture, model scale, and task distribution remain relevant.

Open post-training projects increasingly separate checkpoints and evaluations by stage. Tulu 3 exposes SFT, DPO, and reinforcement-learning artifacts; DeepSeek-R1 documents cold-start SFT, reinforcement learning, rejection-sampled SFT, and later reinforcement learning; and Llama 3 describes repeated rounds of SFT and DPO with synthetic-data generation.^[17][35][36] This stage-level reporting makes it easier to identify which behavior was present after SFT and which appeared only after later optimization.

The central unresolved issue is not whether SFT "works" in the abstract. It is which supervised distribution, objective, trainable parameterization, and evaluation are appropriate for a particular base model and target. Claims about data efficiency, capability gains, safety, cost, or generalization remain conditional until replicated under the relevant protocol.

See also

• Fine Tuning
• Instruction Tuning
• Post-training
• Pre-training
• Supervised Learning
• Transfer Learning
• Parameter-Efficient Fine-Tuning
• LoRA
• Reinforcement Learning from Human Feedback
• DPO
• Knowledge Distillation
• Retrieval-Augmented Generation

References

1. ^Pan, S. J., and Yang, Q. (2009). "A Survey on Transfer Learning." arXiv:0907.1815. arxiv.org/...0907.1815
2. ^Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). "How transferable are features in deep neural networks?" arXiv:1411.1792. arxiv.org/...1411.1792
3. ^Howard, J., and Ruder, S. (2018). "Universal Language Model Fine-tuning for Text Classification." arXiv:1801.06146. arxiv.org/...1801.06146
4. ^Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." arXiv:1810.04805. arxiv.org/...1810.04805
5. ^Raffel, C., Shazeer, N., Roberts, A., et al. (2019). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer." arXiv:1910.10683. arxiv.org/...1910.10683
6. ^Gururangan, S., Marasovic, A., Swayamdipta, S., et al. (2020). "Don't Stop Pretraining: Adapt Language Models to Domains and Tasks." arXiv:2004.10964. arxiv.org/...2004.10964
7. ^Wei, J., Bosma, M., Zhao, V. Y., et al. (2021). "Finetuned Language Models Are Zero-Shot Learners." arXiv:2109.01652. arxiv.org/...2109.01652
8. ^Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. (2021). "Cross-Task Generalization via Natural Language Crowdsourcing Instructions." arXiv:2104.08773. arxiv.org/...2104.08773
9. ^Sanh, V., Webson, A., Raffel, C., et al. (2021). "Multitask Prompted Training Enables Zero-Shot Task Generalization." arXiv:2110.08207. arxiv.org/...2110.08207
10. ^Ouyang, L., Wu, J., Jiang, X., et al. (2022). "Training language models to follow instructions with human feedback." arXiv:2203.02155. arxiv.org/...2203.02155
11. ^Wang, Y., Kordi, Y., Mishra, S., et al. (2022). "Self-Instruct: Aligning Language Models with Self-Generated Instructions." arXiv:2212.10560. arxiv.org/...2212.10560
12. ^Chung, H. W., Hou, L., Longpre, S., et al. (2022). "Scaling Instruction-Finetuned Language Models." arXiv:2210.11416. arxiv.org/...2210.11416
13. ^Longpre, S., Hou, L., Vu, T., et al. (2023). "The Flan Collection: Designing Data and Methods for Effective Instruction Tuning." arXiv:2301.13688. arxiv.org/...2301.13688
14. ^Zhou, C., Liu, P., Xu, P., et al. (2023). "LIMA: Less Is More for Alignment." arXiv:2305.11206. arxiv.org/...2305.11206
15. ^Köpf, A., Kilcher, Y., von Rütte, D., et al. (2023). "OpenAssistant Conversations: Democratizing Large Language Model Alignment." arXiv:2304.07327. arxiv.org/...2304.07327
16. ^Ivison, H., Wang, Y., Pyatkin, V., et al. (2023). "Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2." arXiv:2311.10702. arxiv.org/...2311.10702
17. ^Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). "Tulu 3: Pushing Frontiers in Open Language Model Post-Training." arXiv:2411.15124. arxiv.org/...2411.15124
18. ^Hu, E. J., Shen, Y., Wallis, P., et al. (2021). "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685. arxiv.org/...2106.09685
19. ^Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). "QLoRA: Efficient Finetuning of Quantized LLMs." arXiv:2305.14314. arxiv.org/...2305.14314
20. ^Lialin, V., Deshpande, V., and Rumshisky, A. (2023). "Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning." arXiv:2303.15647. arxiv.org/...2303.15647
21. ^Biderman, D., Portes, J., Gonzalez Ortiz, J. J., et al. (2024). "LoRA Learns Less and Forgets Less." arXiv:2405.09673. arxiv.org/...2405.09673
22. ^Kirkpatrick, J., Pascanu, R., Rabinowitz, N., et al. (2016). "Overcoming catastrophic forgetting in neural networks." arXiv:1612.00796. arxiv.org/...1612.00796
23. ^Kalajdzievski, D. (2024). "Scaling Laws for Forgetting When Fine-Tuning Large Language Models." arXiv:2401.05605. arxiv.org/...2401.05605
24. ^Christiano, P. F., Leike, J., Brown, T., et al. (2017). "Deep Reinforcement Learning from Human Preferences." arXiv:1706.03741. arxiv.org/...1706.03741
25. ^Rafailov, R., Sharma, A., Mitchell, E., et al. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv:2305.18290. arxiv.org/...2305.18290
26. ^Bai, Y., Kadavath, S., Kundu, S., et al. (2022). "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073. arxiv.org/...2212.08073
27. ^Lewis, P., Perez, E., Piktus, A., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." arXiv:2005.11401. arxiv.org/...2005.11401
28. ^Hinton, G., Vinyals, O., and Dean, J. (2015). "Distilling the Knowledge in a Neural Network." arXiv:1503.02531. arxiv.org/...1503.02531
29. ^Qi, X., Zeng, Y., Xie, T., et al. (2023). "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" arXiv:2310.03693. arxiv.org/...2310.03693
30. ^Wan, A., Wallace, E., Shen, S., and Klein, D. (2023). "Poisoning Language Models During Instruction Tuning." arXiv:2305.00944. arxiv.org/...2305.00944
31. ^Hubinger, E., Denison, C., Mu, J., et al. (2024). "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." arXiv:2401.05566. arxiv.org/...2401.05566
32. ^Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. (2023). "What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning." arXiv:2312.15685. arxiv.org/...2312.15685
33. ^Chu, T., Zhai, Y., Yang, J., et al. (2025). "SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training." arXiv:2501.17161. arxiv.org/...2501.17161
34. ^Harada, Y., Yamauchi, Y., Oda, Y., et al. (2025). "Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality." arXiv:2506.14681. arxiv.org/...2506.14681
35. ^DeepSeek-AI. (2025). "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948. arxiv.org/...2501.12948
36. ^Grattafiori, A., Dubey, A., Jauhri, A., et al. (2024). "The Llama 3 Herd of Models." arXiv:2407.21783. arxiv.org/...2407.21783
37. ^Touvron, H., Martin, L., Stone, K., et al. (2023). "Llama 2: Open Foundation and Fine-Tuned Chat Models." arXiv:2307.09288. arxiv.org/...2307.09288
38. ^Li, T., Chiang, W.-L., Frick, E., et al. (2024). "Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators." arXiv:2404.04475. arxiv.org/...2404.04475
39. ^Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." arXiv:2306.05685. arxiv.org/...2306.05685
40. ^Zhou, J., Lu, T., Mishra, S., et al. (2023). "Instruction-Following Evaluation for Large Language Models." arXiv:2311.07911. arxiv.org/...2311.07911
41. ^Chen, J., and Mueller, J. (2024). "Automated Data Curation for Robust Language Model Fine-Tuning." arXiv:2403.12776. arxiv.org/...2403.12776
42. ^Groeneveld, D., Beltagy, I., Walsh, P., et al. (2024). "OLMo: Accelerating the Science of Language Models." arXiv:2402.00838. arxiv.org/...2402.00838
43. ^Fu, Y., Chen, T., Chai, J., et al. (2025). "SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning." arXiv:2506.19767. arxiv.org/...2506.19767
44. ^Jiang, H., Zhang, W., Yao, J., et al. (2026). "Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models." arXiv:2603.13985. arxiv.org/...2603.13985