# SPIN (Self-Play Fine-Tuning)

> Source: https://aiwiki.ai/wiki/spin
> Updated: 2026-06-09
> Categories: AI Alignment, Large Language Models, Training & Optimization
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

# SPIN (Self-Play Fine-Tuning)

**SPIN** (Self-Play fIne-tuNing) is a [post-training](/wiki/post-training) method for [large language models](/wiki/large_language_model) introduced by researchers at the University of California, Los Angeles (UCLA) in January 2024. SPIN iteratively improves a [supervised fine-tuning](/wiki/supervised_fine-tuning) (SFT) checkpoint by framing alignment as a two-player game between the model being trained and a frozen copy of an earlier iteration of itself. The current model learns to distinguish human-written ground-truth responses (treated as "chosen" samples) from responses generated by the previous iteration (treated as "rejected" samples), and the resulting objective can be optimized with the same logistic-loss machinery used by [Direct Preference Optimization (DPO)](/wiki/direct_preference_optimization_dpo).[^1]

The method was proposed in the paper "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models" by Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu, posted to arXiv as 2401.01335 on 2 January 2024 and accepted at the International Conference on Machine Learning ([ICML](/wiki/icml)) 2024.[^1][^2] SPIN demonstrated that an existing SFT checkpoint, the publicly available `zephyr-7b-sft-full` derived from [Mistral 7B](/wiki/mistral_7b), can be significantly improved on the HuggingFace Open LLM Leaderboard, [MT-Bench](/wiki/mt_bench), and [BIG-Bench](/wiki/big_bench) using only the existing SFT dataset (a 50k-prompt subset of UltraChat 200K) plus self-generated synthetic completions, without collecting any additional human preference data.[^1][^3]

## Introduction

By late 2023, the dominant recipes for aligning open-source [LLMs](/wiki/llm) involved either Reinforcement Learning from Human Feedback ([RLHF](/wiki/rlhf)), as popularized for [InstructGPT](/wiki/instructgpt) and [ChatGPT](/wiki/chatgpt), or Direct Preference Optimization ([DPO](/wiki/dpo)), which trains directly on preference pairs without an explicit reward model.[^4][^5] Both approaches share a hard requirement: they need preference data, typically expensive human annotations or, in distillation pipelines, judgements from a stronger model such as GPT-4. The question SPIN addresses is whether a weak SFT model can be pushed substantially further using only the SFT data already on hand.[^1]

The SPIN paper answers this question affirmatively. Starting from the Zephyr 7B SFT checkpoint trained on UltraChat 200K, three iterations of self-play raised the average score on the six tasks of the HuggingFace Open LLM Leaderboard (ARC-Challenge, [HellaSwag](/wiki/hellaswag), [MMLU](/wiki/mmlu), [TruthfulQA](/wiki/truthfulqa), [WinoGrande](/wiki/winogrande), [GSM8K](/wiki/gsm8k)) from 58.14 to 63.16, with the largest absolute gains on GSM8K (26.76 to 38.97) and TruthfulQA (43.73 to 54.90).[^1][^6] The authors also report that SPIN at iteration 1 already surpasses a baseline DPO run on the Zephyr beta preference data (62k pairs of GPT-4-judged completions) on most leaderboard tasks.[^1]

SPIN has been compared to a long tradition of self-play in reinforcement learning, in particular [AlphaGo](/wiki/alphago) and [AlphaZero](/wiki/alphazero), as well as to [generative adversarial networks (GANs)](/wiki/generative_adversarial_network_gan), which similarly pit a generator and a discriminator against one another. A key conceptual difference is that in SPIN both "players" are instantiations of the same language model at different training steps, so no separate discriminator network is required.[^1]

## Background: SFT, RLHF, and DPO

Modern open-source chat models are typically built in three stages. Pretraining yields a base [language model](/wiki/language_model). [Supervised fine-tuning](/wiki/supervised_fine-tuning) on instruction-following demonstrations adapts the base model to the chat format. A preference optimization stage, originally implemented with [PPO](/wiki/ppo)-based [RLHF](/wiki/rlhf) and later with DPO, refines the model on pairs of "chosen" and "rejected" responses.[^4][^5][^7]

[RLHF](/wiki/rlhf) was popularized by the InstructGPT paper of Ouyang et al. (2022), which used a learned reward model trained on human comparison data to optimize a policy via Proximal Policy Optimization ([PPO](/wiki/ppo)).[^4] [DPO](/wiki/dpo), proposed by Rafailov et al. (2023), removes the explicit reward model by reparameterizing the optimal RLHF policy in terms of a closed-form log-ratio and training the language model directly with a binary classification loss on chosen/rejected pairs.[^5] DPO has become a standard component of post-training pipelines such as the Zephyr recipe by Tunstall et al. (2023) at [Hugging Face](/wiki/hugging_face), which reached strong MT-Bench scores by combining SFT on UltraChat with DPO on the GPT-4-judged UltraFeedback dataset.[^7]

A persistent limitation of both [RLHF](/wiki/rlhf) and [DPO](/wiki/dpo) is the cost and availability of preference data. Constructing UltraFeedback-scale datasets requires either human annotators or judgements from frontier proprietary models, and the resulting preference signal is by construction "human-level" in quality.[^7][^8] Methods that try to reduce or eliminate this cost include [Constitutional AI](/wiki/constitutional_ai) and its [RLAIF](/wiki/rlaif) variant, which substitute AI-generated critiques for human preferences,[^9] and self-training approaches such as ReST^EM by Singh, Co-Reyes and colleagues, which iteratively fine-tunes on filtered self-generated solutions to problems with verifiable scalar feedback.[^10] SPIN takes a different angle: it does not generate additional preference labels at all, but rather treats existing ground-truth responses as positives and self-generated continuations as negatives.[^1]

## The SPIN Algorithm

SPIN operates iteratively. Let $\{(x_i, y_i)\}_{i=1}^N$ denote an SFT dataset of prompt-response pairs, where each $y_i$ is the human-written ground-truth completion for prompt $x_i$. Let $p_{\theta_t}$ denote the LLM at iteration $t$, with $\theta_0$ initialized from the SFT checkpoint. SPIN proceeds as follows.[^1]

### Per-iteration procedure

1. **Generation.** For each prompt $x_i$, sample a synthetic response $y'_i \sim p_{\theta_t}(\cdot \mid x_i)$ from the current (frozen) model.
2. **Self-play update.** Solve
$$\theta_{t+1} = \arg\min_{\theta} \; \mathbb{E}_{(x,y) \sim \text{SFT},\, y' \sim p_{\theta_t}(\cdot \mid x)} \Big[ \ell\big( \lambda \log \tfrac{p_\theta(y\mid x)}{p_{\theta_t}(y\mid x)} - \lambda \log \tfrac{p_\theta(y'\mid x)}{p_{\theta_t}(y'\mid x)} \big) \Big]$$
where $\ell$ is a monotonically decreasing convex loss (the paper uses the logistic loss $\ell(t) = \log(1 + e^{-t})$) and $\lambda \ge 0$ is a temperature parameter.[^1]
3. **Repeat** with $t \leftarrow t + 1$, using $p_{\theta_{t+1}}$ to generate the next batch of synthetic negatives.

Intuitively, the main player $p_\theta$ (the current model being trained) is rewarded for raising the log-probability ratio it assigns to the ground-truth response $y$ relative to the frozen previous iteration $p_{\theta_t}$, while simultaneously lowering the same ratio for the self-generated response $y'$.[^1]

### Two-player game formulation

The authors derive the loss from a game-theoretic formulation in which a "main player" $f$ (a real-valued function of $(x, y)$) tries to maximize an Integral Probability Metric (IPM) between the target data distribution $p_\text{data}$ and the opponent distribution $p_{\theta_t}$, while the opponent (the previous LLM iteration) tries to match the data distribution. Restricting the function class of the main player to log-density-ratio functions parameterized by $f(x,y) = \lambda \log [p_\theta(y\mid x) / p_{\theta_t}(y\mid x)]$ yields the SPIN objective. Each iteration thus alternates between two steps of a saddle-point optimization, which the paper analyzes as a fixed-point iteration on the space of LLM policies.[^1]

### Choice of synthetic dataset

In the published experiments, the synthetic prompts are 50,000 random prompts drawn from the HuggingFace `ultrachat_200k` dataset, the same SFT corpus on which `zephyr-7b-sft-full` was trained.[^1][^3] At each iteration, the model generates one synthetic response per prompt with sampling, and the loss is computed on the pair (ground-truth, self-generated). The released codebase at github.com/uclaml/SPIN provides four checkpoints, `iter0` through `iter3`, along with the corresponding parquet datasets.[^3]

## Theoretical Analysis

The SPIN paper provides both a fixed-point characterization of the iterative procedure and an explicit closed-form expression for the per-iteration update under logistic loss.

### Fixed-point characterization (Theorem 5.2)

The central theoretical result is a fixed-point theorem stating, informally, that the SPIN training objective is globally minimized if and only if the model's conditional distribution matches the target data distribution. Concretely, if $p_{\theta_t}(\cdot \mid x) = p_\text{data}(\cdot \mid x)$ then $\theta_t$ is a global minimizer of $L_\text{SPIN}(\theta, \theta_t)$ for any $\lambda \ge 0$; conversely, if $p_{\theta_t}(\cdot \mid x) \ne p_\text{data}(\cdot \mid x)$ then there exists a finite $\lambda$ such that $\theta_t$ is not a global minimizer.[^1] The optimization process therefore "naturally stops" once the model reproduces the empirical data distribution.[^1]

### Closed-form update (Theorem 5.4)

Under logistic loss and an additional regularity condition, the paper derives an idealized expression for the next-iterate distribution:
$$p_{\theta_{t+1}}(y \mid x) \;\propto\; p_{\theta_t}(y \mid x) \left( \frac{p_\text{data}(y \mid x)}{p_{\theta_t}(y \mid x)} \right)^{1/\lambda}.$$
This is a geometric interpolation between the previous policy and the data distribution: as $\lambda \to \infty$, $p_{\theta_{t+1}} \to p_{\theta_t}$; as $\lambda \to 0^+$, $p_{\theta_{t+1}} \to p_\text{data}$.[^1] The expression makes precise the sense in which each SPIN step "moves toward" the data distribution, with $\lambda$ acting as a step size.

### Comparison to GANs and to DPO's KL-regularized objective

The two-player formulation of SPIN echoes the generator/discriminator structure of GANs.[^1][^11] However, SPIN's "discriminator" is constrained to the parametric form of a log density ratio, and both players are the same LLM at different iterations rather than independent networks. This connection has been examined in follow-up work that introduces explicit Kullback-Leibler regularization, equivalent to mixing the previous policy with a base policy ("fictitious play"), to stabilize the self-play dynamics.[^12]

## Experimental Results

### Setup

The base model is `zephyr-7b-sft-full`, the SFT-only checkpoint of HuggingFace's Zephyr recipe, itself initialized from [Mistral 7B](/wiki/mistral_7b) and fine-tuned on UltraChat 200K.[^1][^7] All SPIN experiments share this starting point. Training uses DeepSpeed ZeRO-3, FlashAttention-2, the RMSProp optimizer, bfloat16 precision, and a global batch size of 64 on 8x A100 GPUs, with each iteration running two epochs of supervised optimization over a 50k synthetic dataset.[^3][^6]

### HuggingFace Open LLM Leaderboard

Evaluation uses the standard `lm-evaluation-harness` setup for the HuggingFace Open LLM Leaderboard, which averages six tasks: ARC-Challenge, [HellaSwag](/wiki/hellaswag), [MMLU](/wiki/mmlu), [TruthfulQA](/wiki/truthfulqa), [WinoGrande](/wiki/winogrande), and [GSM8K](/wiki/gsm8k).[^1] The paper reports the following averages and per-task numbers (in percent accuracy, except TruthfulQA which is `mc2` and GSM8K which is exact-match):[^1][^6]

| Model | ARC | TruthfulQA | WinoGrande | GSM8K | HellaSwag | MMLU | Average |
|---|---|---|---|---|---|---|---|
| Baseline (zephyr-7b-sft-full) | 60.41 | 43.73 | 74.19 | 26.76 | 82.85 | 60.92 | 58.14 |
| SPIN iter 0 | 63.40 | 49.18 | 72.69 | 35.10 | 84.38 | 60.03 | 60.80 |
| SPIN iter 1 | 65.19 | 55.17 | 72.30 | 35.78 | 84.96 | 59.34 | 62.12 |
| SPIN iter 3 | 65.87 | 54.90 | 73.72 | 38.97 | 85.54 | 59.99 | 63.16 |

The largest single-iteration gain occurs at iteration 0, with GSM8K improving by more than 8 absolute points and TruthfulQA by more than 5. By iteration 3, the cumulative improvement on average accuracy is approximately 5.02 absolute points, with GSM8K (+12.21) and TruthfulQA (+11.17) carrying most of the gain. Improvements taper after iteration 1 but do not regress.[^1]

### MT-Bench and BIG-Bench

On [MT-Bench](/wiki/mt_bench), the multi-turn LLM-as-judge benchmark of Zheng et al. (2023), the paper reports the SPIN-trained model improving from a baseline of 5.94 to 6.78 by iteration 2, an improvement comparable in magnitude to that obtained by full RLHF on the Zephyr beta model.[^1][^13] Selected [BIG-Bench](/wiki/big_bench) tasks similarly improve across iterations; for instance, Causal Judgment, Formal Fallacies, and Sports Understanding all gain several points relative to the SFT baseline.[^1]

### Comparison to DPO baselines

The most striking experimental claim is that SPIN at iteration 0, using only synthetic negatives, is competitive with DPO trained on the 62k UltraFeedback preference pairs used for `zephyr-7b-beta`, and that SPIN at iteration 1 surpasses DPO on most leaderboard tasks. The paper interprets this as evidence that "iterative training is a necessary component in SPIN as it breaks the limit of multi-epoch training" on a single static preference dataset.[^1] An ablation shows that running DPO for additional epochs on the same 62k preference set does not produce comparable gains.[^1]

## Relation to DPO and Other Methods

### Reduction to DPO loss

A widely cited observation is that the SPIN per-iteration objective, when instantiated with logistic loss, has the same functional form as the DPO loss with the previous iteration $p_{\theta_t}$ serving as the reference policy and the chosen/rejected roles played by ground-truth and self-generated responses respectively.[^1][^14] Concretely, the SPIN loss becomes
$$L_\text{SPIN}(\theta, \theta_t) = - \mathbb{E}\Big[ \log \sigma\big( \lambda \log \tfrac{p_\theta(y\mid x)}{p_{\theta_t}(y\mid x)} - \lambda \log \tfrac{p_\theta(y'\mid x)}{p_{\theta_t}(y'\mid x)} \big) \Big],$$
which is exactly the DPO loss with $\beta = \lambda$, reference policy $\pi_\text{ref} = p_{\theta_t}$, chosen response $y_w = y$, and rejected response $y_l = y'$.[^1][^14]

Despite this equivalence in functional form, the paper emphasizes three differences from standard DPO. First, DPO is typically run as a single offline pass; SPIN runs many iterations, each of which generates its own negatives. Second, DPO requires preference data, whereas SPIN consumes only an SFT dataset and self-generated samples. Third, the SPIN framework permits losses other than logistic, although in practice only the logistic instantiation has been explored experimentally.[^1] The verl reinforcement learning library by ByteDance implements SPIN under this DPO-equivalent view, sharing infrastructure with online DPO.[^14]

### Iterative DPO

Several contemporaneous methods generalize DPO to an iterative or online setting. Snorkel AI's Snorkel-Mistral-PairRM-DPO recipe, released in early 2024, performs three rounds of self-generation followed by DPO using PairRM as a preference oracle.[^15] Methods such as Iterative DPO (Xu et al., 2023; Xiong et al., 2023) and Iterative Reasoning Preference Optimization extend this idea further, using external reward models or rule-based filtering to label self-generated pairs.[^16] SPIN sits within this family as the variant that uses the SFT ground-truth itself, rather than any external preference model, as the positive signal.[^1]

### Self-Rewarding Language Models

A contemporaneous method, "Self-Rewarding Language Models" by Yuan et al. (2024) from Meta, posted to arXiv on 18 January 2024 (sixteen days after SPIN) and accepted at ICML 2024, also performs iterative DPO without external preference data. Self-Rewarding LMs instead use the model itself in an LLM-as-a-Judge role to score its own samples, producing preference pairs that are then used for DPO. Three iterations of self-rewarding on [Llama 2](/wiki/llama_2) 70B reportedly surpass Claude 2, Gemini Pro, and GPT-4-0613 on the AlpacaEval 2.0 benchmark.[^17] SPIN and Self-Rewarding LMs are often discussed together as the two principal "self-improvement via iterative DPO" papers of January 2024; the methods differ in how the rejected response is obtained (a previous iteration of the model, versus a same-iteration sample judged worse by the model-as-judge).[^17]

### Constitutional AI and RLAIF

[Constitutional AI](/wiki/constitutional_ai) and [RLAIF](/wiki/rlaif), introduced by Bai et al. (2022) at Anthropic, similarly aim to reduce reliance on human preference labels by having a model critique and revise its own outputs according to a written constitution, then training a reward model on the resulting AI-generated preferences.[^9] SPIN is more minimal: there is no constitution, no critique, and no separate reward model, and the "preference" is implicit in the contrast between human-written and self-generated text.[^1]

### ReST^EM and self-training

ReST^EM, proposed by Singh, Co-Reyes and collaborators at Google DeepMind in December 2023, performs expectation-maximization-style self-training. The model samples candidate solutions, filters them by a binary correctness signal (for example, whether a math solution matches the gold answer or whether code passes test cases), and fine-tunes on the filtered positives.[^10] Unlike SPIN, ReST^EM requires a verifier and discards rather than penalizes incorrect samples, but both methods iteratively bootstrap an LLM from a fixed seed dataset without additional human labels.[^10]

## Self-Rewarding LMs and Contemporaneous Self-Improvement Methods

The first half of 2024 saw a burst of work on iterative self-improvement of LLMs. Beyond SPIN and Self-Rewarding LMs, notable contemporaneous methods include:

- **Iterative DPO with external reward models.** Snorkel AI's recipe atop `mistral-7b-instruct` and various academic instantiations use a fixed reward model (PairRM, UltraRM) to label self-generated pairs across multiple DPO rounds.[^15]
- **Self-Play Preference Optimization (SPPO).** Introduced by Wu, Sun, Yuan, Ji, Yang, and Gu in May 2024, SPPO frames preference optimization as a constant-sum two-player game and seeks the Nash equilibrium under general preference structures, working directly with preference probabilities rather than Bradley-Terry rewards. On `Mistral-7B-Instruct-v0.2` it reaches a 28.53% length-controlled win rate against GPT-4 Turbo on AlpacaEval 2.0 using only 60k UltraFeedback prompts and the PairRM 0.4B preference model; on Llama-3-8B-Instruct it reaches 38.77%.[^18] SPPO shares authors with SPIN (Yuan, Ji, Gu) and is often viewed as the preference-optimization analogue of SPIN.[^18]
- **SPIN-Diffusion.** Yuan, Chen, Ji, and Gu adapted SPIN to text-to-image diffusion models in February 2024. SPIN-Diffusion exceeds supervised fine-tuning from the first iteration and matches RLHF-based diffusion fine-tuning by iteration two on the Pick-a-Pic dataset, all without paired preference data.[^19]
- **Iterative Reasoning Preference Optimization** by Pang et al. (2024) at Meta combines chain-of-thought generation, rule-based correctness filtering, and DPO to specifically improve mathematical reasoning, illustrating how the iterative-DPO template extends to verifier-driven reasoning tasks.[^16]

## Follow-up Work and Extensions

Within months of release, SPIN attracted both empirical analyses and methodological extensions.

### Regularization and stability

Alami, Abubaker, Achab, Seddik, and Lahlou (2024) examined SPIN's training stability and proposed KL-regularized variants and fictitious-play schemes to dampen oscillations across iterations. They show that KL regularization is equivalent to replacing the previous policy with a geometric mixture of the base and previous policies, and report stabilized MT-Bench and Open LLM Leaderboard gains.[^12]

### SPIN for diffusion models

SPIN-Diffusion, mentioned above, ports the self-play formulation to text-to-image diffusion models, where the "policy" is the score network and "samples" are images.[^19] The same authorial team has also extended self-play ideas to other generative modeling settings.

### Domain-specific adaptations

SPIN has been adapted to specialized tasks where self-generated data quality varies. ExSPIN extends SPIN to text-to-SQL parsing with explicit execution feedback, addressing instability in domains where the SFT ground-truth is itself noisy.[^20] Reports also note that on code generation benchmarks such as SPIDER and BIRD, vanilla SPIN can degrade performance relative to SFT, motivating these task-specific variants.[^20]

### Open-source releases and reproductions

The official `uclaml/SPIN` repository releases the data-generation, training, and evaluation pipelines along with checkpoints for `iter0` through `iter3` of the Zephyr 7B reproduction.[^3] HuggingFace community members and the verl team have integrated SPIN-style training into general-purpose post-training libraries, where it appears as an instantiation of online DPO with a particular choice of negative sampler.[^14]

## Limitations

The SPIN paper and its follow-ups identify several limitations.

**Fixed target distribution.** SPIN converges, by construction, when $p_\theta = p_\text{data}$. The performance ceiling is therefore the distribution of the SFT corpus. If the SFT data is itself imperfect or limited, the SPIN-trained model can be no better than that data on the metrics implicitly captured by the SFT responses.[^1] Self-Rewarding LMs and similar methods attempt to break this ceiling by allowing the target distribution to evolve via self-judging.[^17]

**Iteration-count plateau.** The paper reports the largest gains at iteration 0 and diminishing returns thereafter, with most experiments stopping at iteration 3.[^1] Subsequent analyses report that performance can degrade past a peak iteration on some datasets, motivating regularization or stopping heuristics.[^12]

**Sensitivity to the base model.** SPIN relies on the SFT model being strong enough to produce useful synthetic negatives. On weaker base models or on tasks far from the SFT distribution, the synthetic responses can be too low-quality to provide a useful learning signal, and the iterative procedure can produce negative gains; reported failures include SPIN reducing accuracy by up to 12 percentage points relative to SFT on the SPIDER text-to-SQL benchmark when applied to DeepSeek-Coder 6.7B.[^20]

**Computational overhead.** Each iteration requires generating one synthetic completion per prompt, then performing two epochs of fine-tuning. The compute cost scales linearly with the number of iterations, and the published recipe uses 8x A100 GPUs for the 7B-scale experiments.[^3][^6]

**Distinction from "true" self-play.** Unlike [AlphaGo](/wiki/alphago) or [AlphaZero](/wiki/alphazero), where self-play interacts with an environment that supplies an objective reward, SPIN has no environment and no reward signal. The "win condition" is reproducing the SFT data, which is qualitatively different from improving on an open-ended task.[^1]

**Risk of mode collapse.** The two-player formulation, like other adversarial training paradigms, can suffer instability and oscillation between iterations.[^12]

## See also

- [Direct Preference Optimization (DPO)](/wiki/direct_preference_optimization_dpo)
- [Reinforcement Learning from Human Feedback (RLHF)](/wiki/rlhf)
- [Constitutional AI](/wiki/constitutional_ai)
- [RLAIF](/wiki/rlaif)
- [Supervised fine-tuning](/wiki/supervised_fine-tuning)
- [Mistral 7B](/wiki/mistral_7b)
- [Llama 2](/wiki/llama_2)
- [MT-Bench](/wiki/mt_bench)
- [Synthetic data](/wiki/synthetic_data)
- [AlphaZero](/wiki/alphazero)
- [Generative Adversarial Network (GAN)](/wiki/generative_adversarial_network_gan)

## References

[^1]: Chen, Z., Deng, Y., Yuan, H., Ji, K., & Gu, Q. (2024). "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models." arXiv:2401.01335. https://arxiv.org/abs/2401.01335. Accessed 2026-05-19.

[^2]: International Conference on Machine Learning 2024 accepted papers index entry for "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models." https://openreview.net/forum?id=O4cHTxW9BS. Accessed 2026-05-19.

[^3]: uclaml/SPIN: The official implementation of Self-Play Fine-Tuning (SPIN). GitHub repository README. https://github.com/uclaml/SPIN. Accessed 2026-05-19.

[^4]: Ouyang, L., Wu, J., Jiang, X., et al. (2022). "Training language models to follow instructions with human feedback." arXiv:2203.02155. https://arxiv.org/abs/2203.02155. Accessed 2026-05-19.

[^5]: Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv:2305.18290. https://arxiv.org/abs/2305.18290. Accessed 2026-05-19.

[^6]: UCLAML SPIN project page. "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models." https://uclaml.github.io/SPIN/. Accessed 2026-05-19.

[^7]: Tunstall, L., Beeching, E., Lambert, N., et al. (2023). "Zephyr: Direct Distillation of LM Alignment." arXiv:2310.16944. https://arxiv.org/abs/2310.16944. Accessed 2026-05-19.

[^8]: Cui, G., Yuan, L., Ding, N., et al. (2023). "UltraFeedback: Boosting Language Models with Scaled AI Feedback." arXiv:2310.01377. https://arxiv.org/abs/2310.01377. Accessed 2026-05-19.

[^9]: Bai, Y., Kadavath, S., Kundu, S., et al. (2022). "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073. https://arxiv.org/abs/2212.08073. Accessed 2026-05-19.

[^10]: Singh, A., Co-Reyes, J. D., Agarwal, R., et al. (2023). "Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models." arXiv:2312.06585. https://arxiv.org/abs/2312.06585. Accessed 2026-05-19.

[^11]: Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al. (2014). "Generative Adversarial Networks." arXiv:1406.2661. https://arxiv.org/abs/1406.2661. Accessed 2026-05-19.

[^12]: Alami, R., Abubaker, A., Achab, M., Seddik, M. E. A., & Lahlou, S. (2024). "Investigating Regularization of Self-Play Language Models." arXiv:2404.04291. https://arxiv.org/abs/2404.04291. Accessed 2026-05-19.

[^13]: Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." arXiv:2306.05685. https://arxiv.org/abs/2306.05685. Accessed 2026-05-19.

[^14]: verl documentation, "Recipe: Self-Play Fine-Tuning (SPIN)." https://verl.readthedocs.io/en/latest/algo/spin.html. Accessed 2026-05-19.

[^15]: Snorkel AI, "snorkelai/Snorkel-Mistral-PairRM-DPO" model card on HuggingFace. https://huggingface.co/snorkelai/Snorkel-Mistral-PairRM-DPO. Accessed 2026-05-19.

[^16]: Pang, R. Y., Yuan, W., Cho, K., et al. (2024). "Iterative Reasoning Preference Optimization." arXiv:2404.19733. https://arxiv.org/abs/2404.19733. Accessed 2026-05-19.

[^17]: Yuan, W., Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., & Weston, J. (2024). "Self-Rewarding Language Models." arXiv:2401.10020. https://arxiv.org/abs/2401.10020. Accessed 2026-05-19.

[^18]: Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., & Gu, Q. (2024). "Self-Play Preference Optimization for Language Model Alignment." arXiv:2405.00675. https://arxiv.org/abs/2405.00675. Accessed 2026-05-19.

[^19]: Yuan, H., Chen, Z., Ji, K., & Gu, Q. (2024). "Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation." arXiv:2402.10210. https://arxiv.org/abs/2402.10210. Accessed 2026-05-19.

[^20]: Wang, L., et al. (2024). "ExSPIN: Explicit Feedback-Based Self-Play Fine-Tuning for Text-to-SQL Parsing." Entropy / PubMed Central. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11940967/. Accessed 2026-05-19.