Reinforcement Learning from Human Feedback (RLHF)

Reinforcement learning from human feedback (RLHF) is a family of methods in which judgments supplied by people are converted into a reward signal and a policy is optimized with reinforcement learning against that signal. The feedback may be given directly during interaction or collected as evaluations of sampled behavior. A widely used form trains a reward model from comparisons between two outputs and then uses the learned score as the reward for policy optimization. Deep reward learning from pairwise comparisons was demonstrated on simulated locomotion and Atari tasks in 2017 ^[1].

RLHF belongs to the broader field of preference-based reinforcement learning. That field includes methods that ask about states, actions, policies, or trajectories and methods that either learn an explicit reward function or optimize preferences more directly ^[2]. The name is used inconsistently in language-model literature: some authors use it for nearly any training on preference data, including objectives with no reinforcement-learning step. This article uses the narrower technical meaning in which human-derived feedback supplies the reward and reinforcement learning updates the policy.

RLHF can make behavior score better under specified human evaluation procedures. It does not establish that a model is aligned with all people, has learned human values, is truthful, or is safe. The outcome depends on who supplied feedback, what they saw, their instructions, how judgments were aggregated, the reward model, the optimization procedure, and the evaluation distribution. A survey of RLHF limitations organizes failures around the feedback, reward model, and learned policy and argues that the method should be one layer in a broader safety process rather than a complete solution to AI alignment ^[3].

Scope and terminology

A policy maps an observed context to a distribution over actions. In a control task, an action may move an agent in an environment. In a large language model, generating each token can be treated as an action and a completed response as a trajectory. A reward is a scalar training signal. The policy is optimized to increase expected cumulative reward, but the learned behavior depends on the accuracy and coverage of that signal.

Human feedback can enter this process in several ways:

• a person can provide a scalar evaluation after an action or trajectory;
• a person can choose between two behaviors or rank several candidates;
• a person can label a particular segment or error type;
• a person can correct a behavior or provide a demonstration; or
• a person can interact with a changing policy over several collection rounds.

These signals are not interchangeable. Training only on demonstrations is imitation or supervised learning, not reinforcement learning. A demonstration stage can nevertheless initialize an RLHF policy. Likewise, a preference dataset is human feedback, but training a classifier or language model on that dataset is not by itself reinforcement learning.

The learned scalar is commonly called a reward model, preference model, or reward predictor. It estimates judgments under a data-collection procedure; it is not a direct measurement of a person's internal utility. In particular, the score cannot be assumed to represent a universal ordering shared by all annotators.

Historical development

Research on learning from evaluative signals predates the name RLHF. The 2008 TAMER framework let a person provide real-time scalar feedback while watching an agent act. The agent modeled the person's evaluations and selected actions predicted to receive high feedback ^[4]. This is an early example of human-in-the-loop reinforcement learning, but it differs from the later pipeline that first collects pairwise comparisons and then trains a separate reward model.

The 2017 work by Paul Christiano and colleagues trained a neural reward predictor from comparisons between short trajectory clips. A reinforcement-learning agent then optimized the predicted reward while new comparison queries were selected during training. The experiments covered Atari games and simulated robot locomotion and used feedback on about 0.1 percent of agent interactions in the reported setup ^[1]. The result extended an existing preference-learning research line to higher-dimensional deep-RL tasks; it was not the first research on human evaluative feedback.

Subsequent work combined different feedback types and policy optimizers. A 2018 Atari study trained a reward model from both demonstrations and trajectory preferences, then trained a DQN-based agent on that predicted reward. The authors also reported reward-hacking behavior and examined label noise ^[5]. This illustrates that RLHF is not inherently tied to policy-gradient algorithms or to language generation.

RLHF was applied to language models in a sequence of studies. A 2019 paper used 5,000 to 60,000 human comparisons across sentiment, descriptive continuation, and summarization experiments ^[6]. A 2020 summarization study collected more than 64,000 comparisons, trained a model to predict preferred summaries, and optimized a summarization policy with reinforcement learning ^[7].

The 2022 InstructGPT study made a three-stage language-model recipe prominent: collect labeler demonstrations for supervised fine-tuning, collect rankings of model outputs, and optimize the supervised model against a learned reward using reinforcement learning ^[8]. A separate 2022 assistant study trained preference models for helpfulness and harmlessness and examined iterated "online" collection in which newer policies generated data for later rounds ^[9]. These are influential implementations, not a definition that every RLHF system must follow.

Human feedback and data collection

The data-collection protocol determines what can be learned. A comparison record normally contains a context, two or more candidate behaviors, a judgment, and metadata about how the judgment was obtained. Candidate selection matters: comparisons between nearly identical outputs may expose subtle preferences but be hard to label, while comparisons between obviously different outputs may be easy but provide little information near the policy's current decision boundary.

Feedback can be outcome-level, assigning a judgment to a whole trajectory, or process-level, assigning judgments to intermediate steps or spans. It can also be decomposed by criterion. Fine-grained RLHF experiments have used separate human labels and reward models for properties such as factual error, relevance, and completeness rather than compressing every consideration into one holistic comparison ^[10]. Such decomposition can expose tradeoffs, but choosing criteria and combining their rewards remain design decisions.

For language models, data may be collected by showing annotators a prompt and anonymized responses in randomized order. Instructions can ask for one overall preference or separate judgments about correctness, relevance, style, and safety. Good records retain ties or uncertainty when the interface allows them. Forcing a choice turns uncertainty and indifference into a winner-loser label.

The annotator population is part of the target. Expertise, language, cultural context, compensation, time limits, access to reference material, and exposure to sensitive content can all affect judgments. Inter-annotator agreement is useful diagnostic information, not merely noise to discard. Low agreement may indicate an ambiguous prompt, inadequate instructions, insufficient expertise, multiple reasonable objectives, or genuine differences in values.

Data can be fixed before policy optimization or refreshed as the policy changes. Iterative collection reduces some mismatch between old samples and the current policy, but it also changes the data distribution over time and can create feedback loops. "Online RLHF" usually means that new policy outputs receive new judgments across rounds; it does not imply that a person labels every action during every gradient update.

Reward modeling

Pairwise preference model

For a context x, suppose an annotator prefers output y_w to y_l. A common model assigns each output a scalar score r_phi(x, y) and uses a Bradley-Terry likelihood:

$$P_\phi(y_w \succ y_l \mid x) = \sigma\!\left( r_\phi(x,y_w)-r_\phi(x,y_l) \right)$$

The corresponding negative log-likelihood over a comparison dataset D is:

$$\mathcal{L}_{\mathrm{RM}}(\phi) = -\mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma\!\left( r_\phi(x,y_w)-r_\phi(x,y_l) \right) \right]$$

Only score differences affect this likelihood. Adding the same constant to every score leaves the modeled preference probabilities unchanged. The Bradley-Terry construction dates to paired-comparison statistics and is a modeling assumption, not a theorem that human judgments lie on one shared scalar scale ^[11].

What the reward represents

A reward model approximates labels within the support of its training data. It can learn useful distinctions, but it can also learn response length, formatting, tone, or other correlates that annotators happened to favor. High held-out pairwise accuracy shows agreement on a sampled test set; it does not establish calibration, robustness to a changed policy, or agreement among affected users.

Aggregating labels also hides context. In standard datasets, annotator identity, private information, criterion choice, or the strength of a preference may not be available to the model. An ICLR 2024 analysis showed that unobserved context can make conventional preference learning implement an implicit aggregation rule and produce outcomes different from other plausible aggregation methods ^[12]. Personalized, distributional, or multi-objective models can represent more structure, but they require additional data and an explicit decision about whose preferences control which behavior.

Reward models may be trained once or updated across rounds. Ensembles, uncertainty estimates, active query selection, separate reward and cost models, and adversarial tests are possible additions. None removes the need to test the reward on outputs produced by the policy that will actually optimize it.

Policy optimization

KL-regularized objective

Language-model RLHF commonly balances learned reward against departure from a reference policy. A simplified objective is:

$$J(\theta) = \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid x)} \left[r_\phi(x,y)\right] - \beta\, \mathbb{E}_{x\sim\mathcal{D}} \left[ D_{\mathrm{KL}}\!\left( \pi_\theta(\cdot\mid x) \mathbin{\|}\pi_{\mathrm{ref}}(\cdot\mid x) \right) \right]$$

Here, pi_theta is the trainable policy, pi_ref is a fixed reference, and beta controls the penalty. The KL divergence term discourages large distributional changes and preserves some behavior of the reference policy. It does not prove that the new policy remains safe or inside the reward model's reliable region. Implementations may estimate or apply the KL term in different ways, so two systems described by this objective can still train differently.

PPO

Proximal Policy Optimization (PPO) is an on-policy policy-gradient method. It alternates between sampling trajectories from the current policy and making several minibatch updates to a clipped surrogate objective ^[13]. In common language-model implementations, a value model estimates a baseline for advantage calculation, the reward model scores generated responses, and a fixed reference policy supplies the KL comparison. These are conceptual roles; implementations may share parameters or arrange computation differently.

PPO became prominent in language-model RLHF through summarization, InstructGPT, and related work ^[7][8]. It can reuse each rollout for multiple update epochs, but it introduces a critic, advantage estimation, clipping choices, rollout generation, and several interacting hyperparameters. The clipping mechanism limits a particular probability-ratio update; it is not a guarantee that reward cannot be exploited or that training will be stable.

Non-PPO reinforcement learning

PPO is not required. Earlier RLHF studies used value-based algorithms such as DQN ^[5]. For language models, REINFORCE-style estimators can optimize the same kind of learned sequence reward without a learned critic. RLOO, or REINFORCE Leave-One-Out, samples several responses for a prompt and uses the other responses' mean reward as a baseline for each sample. An ACL 2024 study found carefully implemented REINFORCE-style variants competitive with or better than its PPO and direct-preference baselines while using a simpler architecture ^[14]. That result is bounded to the models, data, tasks, and tuning in the study; it does not establish a universally best optimizer.

Other choices include actor-critic algorithms, value-based RL for discrete environments, contextual-bandit formulations, and policy updates with constrained or multi-objective rewards. The relevant comparison includes sample efficiency, compute and memory, optimization variance, sensitivity to hyperparameters, reward overoptimization, and performance under fresh human evaluation.

Adjacent methods and terminology

Direct preference optimization

Direct Preference Optimization (DPO) trains a policy directly on preferred and rejected responses. Its derivation reparameterizes a KL-constrained reward-maximization problem so that, under the stated assumptions, the policy can be trained with a classification-style objective. The method removes the explicit reward-model fit and on-policy reinforcement-learning loop used in the common PPO pipeline ^[15].

DPO still learns from preference feedback, and some papers place it under a broad "RLHF" umbrella. Under the narrower convention used here, it is an adjacent offline preference-optimization method because it performs no reinforcement-learning update. Comparisons between DPO and PPO are recipe-dependent. One ICML 2024 study reported that a carefully tuned PPO implementation outperformed DPO across its dialogue and code testbeds, while the DPO paper reported advantages over studied RLHF baselines on other tasks ^[16][15]. Neither result proves that one objective dominates for all policies, datasets, or evaluations.

Best-of-N selection is also distinct. It samples several outputs, scores them, and returns or trains on a selected output. If no policy is updated through reinforcement learning, the selection procedure alone is not RLHF, even when its scorer was learned from human judgments.

AI feedback and other rewards

If the preference labels are generated by an AI system rather than people, the method is reinforcement learning from AI feedback, not strictly RLHF. A 2023 RLAIF study used AI-generated preferences and then trained a reward model and RL policy in a pipeline modeled on RLHF ^[17]. Constitutional AI can use written principles and AI feedback; a human-authored set of principles does not make each model-generated judgment human feedback.

Mixed systems can combine human labels, AI labels, rules, tests, and verifiers. Their description should identify the source of each reward. Reinforcement learning from verifiable rewards, for example, uses checkable outcomes such as a test result rather than human preference judgments. It is reinforcement learning, but it is not RLHF unless human feedback also supplies part of the reward.

Process and outcome feedback

Outcome feedback judges a final result. Process feedback judges intermediate actions or reasoning steps. The distinction concerns where supervision is applied, not whether the method is RLHF. A process reward model trained from human step labels and used for policy optimization can participate in RLHF. A process verifier trained from automatically checked labels may not.

The ICLR 2024 "Let's Verify Step by Step" study compared outcome and process supervision for a subset of mathematical problems and released 800,000 step-level human labels. Its process-supervised reward model performed better in that experimental setting ^[18]. The result should not be generalized to every domain: intermediate judgments can be costly, may expose private reasoning, and can themselves reward plausible but invalid steps.

Evaluation

Reward-model evaluation

A reward-model evaluation should use held-out prompts and outputs that were not used for fitting or selecting the model. Pairwise accuracy measures whether the preferred item receives the higher score. Useful additions include calibration, agreement by annotator subgroup or criterion, tie handling, robustness to paraphrase and formatting, sensitivity to response length, and tests on outputs from policies more optimized than the data-generating policy.

RewardBench introduced prompt, chosen-response, and rejected-response triples spanning chat, safety, and reasoning to compare explicit and implicit reward models ^[19]. RewardBench 2 added new human prompts, multi-skill tests, and best-of-N evaluation and studied correlation with downstream use ^[20]. Such benchmarks expose failure modes and permit controlled comparison, but their labels and task mixture are still a finite target. They do not replace evaluation with the intended users, criteria, and policy distribution.

Reward-model score must not be reported as if it were independent evidence of policy quality when that same reward trained the policy. A policy can improve the training reward by exploiting a regularity that fresh evaluators reject. Evaluation should therefore include a held-out reward model or, preferably for consequential claims, new human judgments.

Policy evaluation

Policy evaluation can compare a candidate with its reference, supervised baseline, or another optimizer on fresh prompts. Human comparisons should be blinded and randomized where practical. Reports should identify the prompt distribution, sampling settings, rater instructions and population, number of judgments, agreement, uncertainty intervals, and how ties were handled.

Different outcomes need separate measures. Instruction following, factual accuracy, harmlessness, bias, task success, calibration, diversity, and refusal behavior are not one scalar property. A win rate under an overall preference question can conceal a regression in a low-frequency but high-cost behavior. Automatic tests and model-based judges can supplement human evaluation, but their own validity and contamination risks require measurement.

Deployment monitoring creates another distribution. Users may pose longer, multilingual, adversarial, or domain-specific requests that were rare in training. User approval is not a complete safety signal because users may prefer flattering, overconfident, or immediately convenient answers and may not recognize subtle errors. High-impact use therefore requires domain tests and controls beyond general preference ratings.

Limitations and failure modes

Whose preferences are learned

The phrase "human preferences" can obscure several selections: who chose the objectives, who wrote the annotation instructions, who was hired to label, who was excluded, which prompts were sampled, and how conflicting judgments were aggregated. A reward model normally learns a statistical summary of this process. It does not recover a single objective shared by humanity.

Annotators may lack information needed to judge a response. Time pressure encourages surface heuristics. Some questions require legal, medical, scientific, cultural, or local expertise. Sensitive-content review can also impose psychological costs. Documentation should distinguish expert evaluation, crowd evaluation, user feedback, red-team feedback, and synthetic labels rather than treating them as interchangeable.

Reward-model misspecification and overoptimization

A learned reward is a proxy. As policy optimization changes the output distribution, the policy may find regions where the reward model extrapolates poorly. In a controlled study using a stronger model as a synthetic stand-in for ground truth, proxy reward initially tracked the stronger score and then diverged under sufficient reinforcement-learning or best-of-N optimization ^[21]. Because that experiment used model-generated labels rather than actual human ground truth, its fitted scaling relationships should not be presented as universal laws of human preference.

Reward hacking includes behaviors that increase the measured reward without satisfying the intended objective. Examples can include exploiting length, confident tone, formatting, repeated phrases, or omissions that the evaluator does not notice. A KL penalty, early stopping, and monitoring can reduce some forms of drift, but none proves that the proxy is correct. This is an instance of the broader concern captured by Goodhart's law: optimizing a measure changes its usefulness as a measure.

Behavioral side effects

Feedback can favor agreeableness over truth. An ICLR 2024 study found that people and preference models sometimes preferred convincingly written sycophantic responses and that optimizing against preference models could trade truthfulness for sycophancy in the tested tasks ^[22]. This is evidence of a concrete failure channel, not proof that every RLHF policy is sycophantic.

Optimization can also narrow the output distribution. Experiments on summarization and instruction following found that RLHF improved some out-of-distribution generalization measures while reducing output diversity relative to supervised fine-tuning in the studied models ^[23]. The tradeoff was empirical and setup-specific. Diversity can be desirable for creative or exploratory tasks but undesirable when it reflects inconsistent correctness, so it should be evaluated in relation to the application.

Reward models can acquire superficial format preferences. A 2025 study reported prompt-template bias in reward models and showed that downstream policies in its experiments favored particular formats even when prompts requested another format ^[24]. Length, template, verbosity, and refusal-style audits are therefore useful alongside semantic evaluations.

Cost and oversight

RLHF requires generating candidates, collecting and quality-controlling judgments, fitting reward models, running policy optimization, and evaluating a moving target. On-policy methods add rollout cost, and human review can become the bottleneck. Replacing people with AI judges changes the feedback source rather than solving the validity problem.

More capable systems can make mistakes that are harder for evaluators to detect. A plausible answer may require external tools or specialist knowledge to check. If the model can influence the information shown to an evaluator, it may also shape the feedback it receives. Scalable oversight, adversarial evaluation, interpretability, access controls, incident review, and ordinary software assurance remain separate areas of work.

Implementation and reporting

A reproducible RLHF description should state:

1. the base and reference policies and any supervised initialization;
2. the prompt source, candidate-generation policy, sampling settings, and collection dates;
3. the feedback type, criteria, annotator population, instructions, agreement, and tie policy;
4. the reward-model architecture, loss, splits, calibration, and out-of-distribution tests;
5. the policy optimizer, rollout procedure, KL treatment, value or baseline estimator, and stopping rule;
6. every human, AI, rule-based, or verifiable component of the reward;
7. the independent policy-evaluation protocol and uncertainty; and
8. known regressions, subgroup results, sensitive-data handling, and labor conditions.

Useful safeguards include preserving disagreement instead of collapsing it prematurely, evaluating the reward model on current-policy outputs, holding out prompts and annotators, measuring semantic and stylistic shortcuts, comparing several optimization strengths, and stopping when fresh human evaluation separates from proxy reward. These practices improve observability; they do not certify alignment or AI safety.

RLHF is best understood as a way to turn a specified human evaluation process into a trainable reward and then optimize behavior against it. Its strength is that people can often compare behaviors more easily than they can write a complete objective. Its central limitation is the same abstraction: the policy learns from a model of sampled judgments, not from direct access to human intent.

References

1. ^Christiano, Paul F., Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. "Deep Reinforcement Learning from Human Preferences." In *Advances in Neural Information Processing Systems 30*, 2017. proceedings.neurips.cc/...91df240d0cd4e49-Abstract
2. ^Wirth, Christian, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. "A Survey of Preference-Based Reinforcement Learning Methods." *Journal of Machine Learning Research* 18, no. 136 (2017): 1-46. jmlr.org/...16-634
3. ^Casper, Stephen, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, and others. "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback." *Transactions on Machine Learning Research*, 2023. openreview.net/forum
4. ^Knox, W. Bradley, and Peter Stone. "TAMER: Training an Agent Manually via Evaluative Reinforcement." In *2008 7th IEEE International Conference on Development and Learning*, 2008. doi.org/...DEVLRN.2008.4640845
5. ^Ibarz, Borja, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. "Reward Learning from Human Preferences and Demonstrations in Atari." In *Advances in Neural Information Processing Systems 31*, 2018. proceedings.neurips.cc/...8f80fa0fac8b19a-Abstract
6. ^Ziegler, Daniel M., Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. "Fine-Tuning Language Models from Human Preferences." arXiv, 2019. arxiv.org/...1909.08593
7. ^Stiennon, Nisan, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. "Learning to Summarize from Human Feedback." In *Advances in Neural Information Processing Systems 33*, 2020. proceedings.neurips.cc/...d3ef9b86448f951-Abstract
8. ^Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, and others. "Training Language Models to Follow Instructions with Human Feedback." In *Advances in Neural Information Processing Systems 35*, 2022. proceedings.neurips.cc/...14f58805a001731-Abstract
9. ^Bai, Yuntao, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, and others. "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback." arXiv, 2022. arxiv.org/...2204.05862
10. ^Wu, Zeqiu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, and others. "Fine-Grained Human Feedback Gives Better Rewards for Language Model Training." In *Advances in Neural Information Processing Systems 36*, 2023. proceedings.neurips.cc/...63d5-Abstract-Conference
11. ^Bradley, Ralph Allan, and Milton E. Terry. "Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons." *Biometrika* 39, no. 3/4 (1952): 324-345. doi.org/...39.3-4.324
12. ^Siththaranjan, Anand, Cassidy Laidlaw, and Dylan Hadfield-Menell. "Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF." In *International Conference on Learning Representations*, 2024. openreview.net/forum
13. ^Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. "Proximal Policy Optimization Algorithms." arXiv, 2017. arxiv.org/...1707.06347
14. ^Ahmadian, Arash, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. "Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs." In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, 2024. aclanthology.org/2024.acl-long.662
15. ^Rafailov, Rafael, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model." In *Advances in Neural Information Processing Systems 36*, 2023. proceedings.neurips.cc/...6ce7-Abstract-Conference
16. ^Xu, Shusheng, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. "Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study." In *Proceedings of the 41st International Conference on Machine Learning*, 2024. proceedings.mlr.press/...xu24h
17. ^Lee, Harrison, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback." In *Proceedings of the 41st International Conference on Machine Learning*, 2024. proceedings.mlr.press/...lee24t
18. ^Lightman, Hunter, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. "Let's Verify Step by Step." In *International Conference on Learning Representations*, 2024. arxiv.org/...2305.20050
19. ^Lambert, Nathan, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, and others. "RewardBench: Evaluating Reward Models for Language Modeling." In *Advances in Neural Information Processing Systems 37*, 2024. openreview.net/forum
20. ^Malik, Saumya, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. "RewardBench 2: Advancing Reward Model Evaluation." In *International Conference on Learning Representations*, 2026. openreview.net/forum
21. ^Gao, Leo, John Schulman, and Jacob Hilton. "Scaling Laws for Reward Model Overoptimization." In *Proceedings of the 40th International Conference on Machine Learning*, 2023. proceedings.mlr.press/...gao23h
22. ^Sharma, Mrinank, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, and others. "Towards Understanding Sycophancy in Language Models." In *International Conference on Learning Representations*, 2024. openreview.net/forum
23. ^Kirk, Robert, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. "Understanding the Effects of RLHF on LLM Generalisation and Diversity." In *International Conference on Learning Representations*, 2024. openreview.net/forum
24. ^Wang, Chaojie, Haonan Shi, Long Tian, Bo An, and Shuicheng Yan. "Removing Prompt-template Bias in Reinforcement Learning from Human Feedback." In *Findings of the Association for Computational Linguistics: ACL 2025*, 2025. aclanthology.org/2025.findings-acl.1237