Citation and evidence

Post-training

25 min full readUpdated 33 references

This article's verification

Report a problem with this article

Edit article
More

Cite this article

Post-training is the stage of large language model (LLM) development that comes after pre-training and turns a raw, general-purpose base model into an aligned, instruction-following AI assistant. It groups together the techniques applied to a model after large-scale pre-training, principally supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and preference optimization such as Direct Preference Optimization (DPO).[1] Post-training has historically used far less compute than pre-training, but it largely determines whether a model is useful in practice, and with reasoning models trained by reinforcement learning its share of the compute budget has grown.[4]

The decisive evidence for post-training's leverage came from OpenAI's 2022 InstructGPT result: human labelers preferred the outputs of a 1.3-billion-parameter post-trained model over those of the 175-billion-parameter GPT-3 base model, despite the post-trained model having 100 times fewer parameters.[7] The field has evolved rapidly from 2022's RLHF breakthrough with InstructGPT, to 2023's DPO simplification of the RLHF pipeline, to the 2024-2025 reasoning models (OpenAI o1, DeepSeek-R1) trained with reinforcement learning on verifiable rewards.[4][29]

What is post-training?

Post-training refers to the collection of processes and techniques applied to a model after its initial, large-scale pre-training phase. According to DeepLearning.AI, post-training "transforms a general-purpose token predictor, trained on trillions of unlabeled text tokens, into an assistant that follows instructions and performs specific tasks."[2] This crucial stage transforms a general-purpose foundation model into a specialized, efficient, and aligned tool ready for real-world deployment.[5]

The PyTorch Foundation defines post-training (sometimes called "alignment") as "a key component of modern LLMs, and the way to 'teach' models how to answer in a way that humans like, and how to reason."[1] This phase addresses fundamental limitations that emerge from pre-training alone:

  • Base models don't naturally understand conversational structure

  • Can generate toxic or harmful content

  • Produce confident hallucinations

  • Lack awareness of when to refuse inappropriate requests

Why is post-training necessary?

Pre-trained architectures reveal significant limitations in specialized contexts, including restricted reasoning capacities, ethical uncertainties, and suboptimal domain-specific performance.[6] OpenAI's InstructGPT research demonstrated this dramatically: their 1.3B parameter post-trained model was preferred by humans over the 175B parameter GPT-3 base model (despite being 100 times smaller) because post-training unlocked capabilities that prompt engineering alone couldn't elicit.[7] The InstructGPT authors concluded that "fine-tuning with human feedback is a promising direction for aligning language models with human intent."[7]

Post-training in the development pipeline

Modern LLM development follows a structured sequence:

  1. Pre-training (weeks to months, consuming millions of GPU-hours on trillions of tokens) builds general language understanding

  2. Post-training (days to weeks, using thousands to millions of curated examples) develops instruction-following, alignment, and specialized capabilities

Recent models employ sophisticated multi-stage approaches: Llama 3 used three pre-training stages (15.6T core tokens, 800B context extension tokens, 40M annealing tokens) followed by multiple post-training rounds combining supervised fine-tuning, rejection sampling, and Direct Preference Optimization.[3]

Original diagram of an RLHF example: pretrained model, supervised fine-tuning, reinforcement learning with a learned reward model, and a post-trained model.

One RLHF-based instruction-tuning path starts from a pretrained model, applies supervised fine-tuning on demonstrations, then uses reinforcement learning with a learned reward model. This simplified sequence is not mandatory for all post-training methods and omits reward-model training details.[32] Illustration: AI Wiki original diagram. CC0 1.0.

How does pre-training differ from post-training?

Pre-training and post-training are complementary phases in the development of modern AI models, especially large models. Pre-training is the process where a model learns from a very large dataset to acquire general knowledge or representations, often without any task-specific supervision. Post-training occurs after this initial learning and focuses on specialization and refinement for particular objectives.

AspectPre-trainingPost-training
PurposeLearn general patterns and representations from large-scale dataRefine and adapt the model for specific tasks, objectives, or constraints
DataVast, diverse, often unlabeled datasets (for example Common Crawl text, ImageNet)Smaller, focused datasets tailored to target task or domain
Duration & ComputeMost resource-intensive phase, requiring extensive computation (large GPU/TPU clusters) and time (weeks to months for frontier models)Shorter and less costly than pre-training; ranges from hours for small fine-tunes to weeks for frontier post-training
OutcomeA general-purpose model (foundation model) that can be adapted to various tasksA task-optimized model tuned for particular application with improved performance and alignment
Generalization vs. SpecializationEmphasizes broad generalization across many tasks and domainsEmphasizes high performance on specific target task(s)
FrequencyUsually done once to create a base modelCan be done multiple times or continuously as new data or requirements emerge

Expanded article table

Core techniques

Supervised fine-tuning

Supervised Fine-Tuning (SFT) serves as the foundational post-training technique, training models on high-quality input-output pairs where responses have been verified beforehand. The PyTorch primer describes SFT's focus as "imitation," teaching the model to learn ideal responses step by step through structured examples.[1]

The technical process resembles pre-training but with a critical difference: loss is calculated only on response tokens, not prompts. Training data follows the format (system_prompt, user_input, ideal_response), and while the entire sequence passes through the model for context, gradient updates occur only on the assistant's response portion.

How much data does SFT need?

ModelSFT ExamplesEpochs
InstructGPT~13,000 prompts16
Qwen 2500,0002
Llama 3.1Not disclosed (largely synthetic, rejection-sampled)Multiple on some sources

Expanded article table

OpenAI's InstructGPT used approximately 13,000 training prompts with human-written demonstrations for SFT, a tiny fraction compared to pre-training datasets.[7] More recent models use larger SFT datasets, with synthetic data generation using larger teacher models becoming standard practice.

Reinforcement learning from human feedback

RLHF represents a paradigm shift in post-training, using human preferences as reward signals to align model behavior with complex human values difficult to specify algorithmically.[8] The technique follows a three-stage process pioneered by OpenAI's InstructGPT paper (arXiv:2203.02155), published 4 March 2022:[7]

  1. Stage one: Supervised fine-tuning on human demonstrations establishes a baseline instruction-following model

  2. Stage two: Training a reward model by collecting human preference data; labelers compare multiple model outputs for the same prompt and rank them

  3. Stage three: Reinforcement learning optimization using algorithms like Proximal Policy Optimization (PPO)

InstructGPT collected roughly 33,000 prompts with human preference comparisons, training a reward model to predict which outputs humans prefer using the Bradley-Terry preference model.[7]

PPO mechanics

The algorithm uses importance sampling to compare new and old policy outputs, clips the ratio to prevent excessive updates (typically within 1±ϵ1 \pm \epsilon where ϵ=0.2\epsilon = 0.2), and combines this with advantage estimation, value function training, and entropy bonuses for exploration.[1] PPO achieves stability through a clipped surrogate objective function. At each update step, it compares the probability of an action under the new policy to that of the old policy. If this ratio becomes too large or too small, the objective function "clips" the update, preventing large, destabilizing changes.

Infrastructure demands

RLHF requires running multiple models simultaneously:

  • The policy model generating responses

  • A reference model computing KL penalties

  • The reward model scoring outputs

  • Optionally a critic network for value estimation

This creates significant computational overhead. At ecosystem scale, NVIDIA has estimated that developing the hundreds or thousands of derivative models built from a popular open model could take around 30x more compute than pretraining the original foundation model.[9]

Direct preference optimization

Direct Preference Optimization (DPO) emerged in May 2023 as a transformative simplification of RLHF, eliminating the need for explicit reward models and reinforcement learning while achieving comparable or superior performance.[10] In the paper "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model," Rafailov et al. of Stanford University showed that language models can implicitly represent reward models, enabling direct optimization through a simple classification loss.[10] The authors reported that DPO "exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train."[10]

Technical innovation

DPO reformulates the RLHF objective into supervised learning using preference pairs: (prompt, preferred_response, rejected_response). The loss function maximizes the log-odds ratio between preferred and rejected responses while maintaining KL divergence constraints to a reference model.[10]

The DPO loss function is formulated as:

LDPO(πθ;πref)=−E(x,yw,yl)∼D[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]

Where:

  • {\displaystyle x} is the prompt, {\displaystyle y_{w}} is the winning (chosen) response, and {\displaystyle y_{l}} is the losing (rejected) response

  • {\displaystyle \pi _{\theta }} is the policy model being trained

  • {\displaystyle \pi _{\text{ref}}} is a fixed reference model (usually the initial SFT model)

  • {\displaystyle \beta } is a hyperparameter that controls how much the policy should deviate from the reference model

  • {\displaystyle \sigma } is the sigmoid function

The computational advantages are substantial:

  • Single forward pass per preference pair (vs multiple models for PPO)

  • Training stability improves dramatically

  • Implementation possible in under 200 lines of code

Adoption and limitations

Since 2023, DPO has become the most popular RLHF alternative, especially in open-source communities. Meta's Llama 3.1 explicitly chose DPO over PPO, finding it more stable and easier to scale.[3] Hugging Face's TRL library, Axolotl, and other major frameworks provide comprehensive DPO support.

Constitutional AI

Anthropic's Constitutional AI represents a fundamental rethinking of alignment, replacing extensive human feedback with AI-generated feedback guided by explicit constitutional principles.[11] This approach enables scalable oversight while maintaining transparency about value systems encoded in models.

Two-phase methodology

  1. Supervised learning phase (SL-CAI): The model critiques its own responses using constitutional principles, then revises responses to align with principles

  2. Reinforcement learning phase (RL-CAI or RLAIF): AI evaluators apply constitutional principles to generate preference data for training

Constitution design

Anthropic's May 2023 constitution drew from diverse sources:[12]

  • UN Declaration of Human Rights

  • Apple's Terms of Service

  • DeepMind's Sparrow principles

  • Non-Western philosophical traditions

  • Anthropic's safety research

The 2023 list of principles covered helpfulness, harmlessness, honesty, and specific values like privacy protection and non-discrimination. In January 2026 Anthropic replaced it with a new, much longer constitution that explains the reasoning behind Claude's values rather than listing standalone rules, and released it under a CC0 license.[33]

Preference optimization variants

Beyond standard DPO and RLHF, the field has rapidly developed specialized preference optimization methods:

MethodKey InnovationUse Case
IPOIdentity function regularizationPrevents overfitting
KTOBinary labels instead of pairsSimpler data collection
ORPOSingle-stage trainingMemory efficiency
GRPONo critic network neededReasoning RL (math, code)

Expanded article table

Kahneman-Tversky Optimization (KTO) draws from behavioral economics and prospect theory to align LLMs, requiring only binary labels (desirable/undesirable) rather than paired comparisons.[13]

Odds Ratio Preference Optimization (ORPO) combines SFT with preference optimization in a unified loss, improving efficiency and performance on benchmarks.[14]

Group Relative Policy Optimization (GRPO), introduced by DeepSeek in 2024, removes the value/critic network of PPO and estimates the baseline from a group of sampled outputs, making it more memory-efficient. It was used to train DeepSeek-R1.[15][29]

Model optimization and compression

Beyond alignment, a major goal of post-training is to optimize models for efficient deployment. This field, known as model compression, aims to reduce a model's size, memory footprint, and computational requirements without significantly degrading its performance.[16]

Post-training quantization

Quantization is a widely used compression technique that reduces the numerical precision of a model's parameters (weights) and/or intermediate calculations (activations).[5] Most neural networks are trained using 32-bit floating-point numbers (FP32). Quantization converts these values to lower-precision formats, such as 16-bit floats (FP16), 8-bit integers (INT8), or even 4-bit integers.

The primary benefits are:

  • Reduced Model Size: Converting from FP32 to INT8 can reduce the model size by approximately 4x

  • Faster Inference: Integer arithmetic is significantly faster than floating-point arithmetic on most CPUs and specialized hardware accelerators

  • Lower Energy Consumption: Faster computation and reduced memory access lead to lower power usage

Post-Training Quantization (PTQ)

Post-Training Quantization (PTQ) is applied to a model that has already been fully trained. It is a popular choice because it is fast and does not require access to the original training dataset or an expensive retraining process. There are two main types of PTQ:

  • Post-Training Dynamic Quantization: The model's weights are quantized offline, but the activations are quantized "on the fly" during inference. This method is simple to apply but adds computational overhead at runtime.

  • Post-Training Static Quantization: Both weights and activations are quantized offline using a small calibration dataset (typically a few hundred representative samples). This generally results in lower inference latency than dynamic quantization.

Methods include:

  • Min-Max Calibration: Fast but outlier-sensitive

  • Activation-Aware Weight Quantization (AWQ): Prioritizes salient weights

  • SmoothQuant: Handles activation outliers

The AWQ authors report more than 3x speedup over FP16 inference with their TinyChat runtime, along with reduced memory usage.[17]

Pruning

Pruning is a model compression technique based on the observation that many deep neural networks are highly over-parameterized, containing redundant weights and neurons that contribute little to their overall performance.[18] Pruning systematically removes these unimportant parameters to create a smaller, more computationally efficient model.

Pruning methodologies

  • Unstructured Pruning: Removes individual weights based on importance scores, resulting in sparse weight matrices

  • Structured Pruning: Removes entire groups of related parameters (filters, channels, attention heads, neurons, or layers), maintaining dense matrix structure for efficient hardware execution

The most effective approach is often iterative pruning:

  1. Train a dense model

  2. Prune a certain percentage of the least important parameters

  3. Fine-tune the pruned model to recover any accuracy lost during pruning

  4. Repeat steps 2 and 3 until desired sparsity is achieved

Knowledge distillation

Knowledge distillation is a compression technique that involves transferring the "knowledge" from a large, complex, and high-performing model (the teacher) to a smaller, more efficient model (the student).[19]

Types of knowledge transfer

  • Response-based (Logit) Distillation: The student model is trained to match the full probability distribution produced by the teacher's output layer (soft labels)

  • Feature-based (Hint) Distillation: The student is trained to mimic the teacher's intermediate feature representations from hidden layers

These compression techniques are often most powerful when used in combination; for example, using knowledge distillation followed by structured pruning and finally post-training static quantization.

TechniqueCore IdeaPrimary BenefitMain DrawbackBest For...
QuantizationReduce numerical precision of weights/activationsSignificant size reduction and faster inferencePotential accuracy degradation at low bit-widthsDeploying on resource-constrained hardware
PruningRemove redundant parametersReduces model complexity and FLOPsHardware acceleration challenges for unstructured pruningReducing latency where some accuracy trade-off acceptable
Knowledge DistillationTrain smaller model to mimic larger oneCompresses knowledge into smaller architectureRequires powerful teacher model and full training cycleCreating compact models for specific tasks

Expanded article table

When did post-training emerge?

Early foundations (2008-2020)

The roots of post-training extend back to early work on learning from human feedback in reinforcement learning. The foundational breakthrough came in June 2017 with "Deep Reinforcement Learning from Human Preferences" by Christiano et al. at OpenAI and DeepMind, demonstrating that learning from pairwise human comparisons could train reward models for complex RL tasks.[20]

Instruction tuning emerges (2021-2022)

Google's FLAN (Finetuned Language Net) paper in 2021 introduced instruction fine-tuning at scale, training models on diverse tasks with natural language instructions.[21] March 2022 marked a watershed moment with OpenAI's InstructGPT paper, establishing the standard three-stage RLHF pipeline still widely used today.[7]

December 2022 saw Anthropic's Constitutional AI paper introducing RLAIF (RL from AI Feedback), pioneering AI-driven alignment techniques.[11]

Direct preference methods (2023-2024)

May 2023 brought another paradigm shift with Direct Preference Optimization by Rafailov et al. at Stanford University.[10] DPO's key insight, that language models implicitly represent reward models, enabled eliminating the explicit reward model and RL training loop.

Reasoning models era (2024-2025)

The frontier of post-training shifted dramatically toward reasoning capabilities in late 2024 and 2025. On 12 September 2024, OpenAI's o1 model introduced "thinking" modes where models engage in extended chain-of-thought reasoning before generating final answers; OpenAI described o1 as "a new large language model trained with reinforcement learning to perform complex reasoning."[31] DeepSeek-R1, released in January 2025 (arXiv:2501.12948), provided the first widely studied open reproduction of reasoning-model training, using GRPO for efficient reinforcement learning and achieving 79.8% on AIME 2024 and 97.3% on MATH-500, rivaling o1.[4][29] DeepSeek reported that its R1-Zero variant developed reasoning behaviors "without any supervised data," relying purely on reinforcement learning.[29]

Post-training teaches LLMs to reason beyond prediction through techniques like:

  • Chain-of-Thought (CoT): Prompts step-by-step thinking, improving accuracy (for example PaLM 540B from 18% to 57% on GSM8K)

  • Tree-of-Thought (ToT): Branches reasoning paths, solving 74% of puzzles vs. 4% with CoT

  • Reflexion: Self-reflects on errors, raising success from 80% to 91% on HumanEval

  • Retrieval-Augmented Generation (RAG): Grounds responses in retrieved data, reducing hallucinations

Industry adoption

Major companies

CompanyModelPost-training Method
OpenAIChatGPT/GPT-4RLHF with PPO
AnthropicClaudeConstitutional AI
Google DeepMindGeminiRLHF + SFT
MetaLlamaDPO + Rejection Sampling
MicrosoftGitHub CopilotRLHF (via OpenAI)

Expanded article table

OpenAI

OpenAI pioneered RLHF for language models with InstructGPT in March 2022, establishing the three-step process of supervised fine-tuning, reward model training, and PPO-based reinforcement learning.[7] OpenAI reported in February 2026 that ChatGPT had 900 million weekly active users.[22]

Anthropic

Anthropic developed Constitutional AI as its core alignment methodology, using AI-generated feedback guided by explicit principles rather than extensive human feedback.[11] Claude model evolution runs from Claude 1 (March 2023; its context window grew from about 9K to 100K tokens that May) through the Claude 4 family (from May 2025) and later releases.

Google DeepMind

Google DeepMind's Gemini family demonstrates sophisticated multimodal post-training, combining RLHF, supervised fine-tuning, and safety filtering.[23] Gemini 1.5 Pro offered a context window of up to 2 million tokens in 2024.

Meta

Meta's Llama model family represents the most transparent post-training implementations. Llama 3.1's post-training used six iterative rounds of supervised fine-tuning, rejection sampling, and DPO.[3] Meta explicitly avoided PPO-based RLHF, finding DPO more stable and easier to scale.

Applications

Conversational AI

  • ChatGPT: 900M weekly active users (February 2026), RLHF with PPO

  • Claude: Constitutional AI, context windows of 200K tokens or more

  • Gemini: Multimodal integration across Google services

Post-training is essential for transforming base models into helpful conversational agents. Models like ChatGPT and Claude undergo extensive post-training using SFT and RLHF to transform them from simple text predictors into helpful, harmless, and instruction-following assistants.[24]

Code generation

GitHub Copilot revolutionized software development through post-trained code models, providing real-time code completion across multiple programming languages. Post-training with feedback from developers helps the model learn what constitutes "good" code in practice.

Domain-specific systems

Medical AI demonstrates post-training's power in specialized domains. Google's MedGemma underwent domain-specific post-training on medical imaging data including chest X-rays, histopathology, and dermatology images.[25]

Finance: Models are fine-tuned on financial news, earnings reports, and market data to perform specialized tasks like sentiment analysis, risk assessment, or algorithmic trading.

Multimodal models

Post-training is critical for text-to-image models like DALL-E and Midjourney. While pre-training teaches them the association between text and images, post-training using human feedback on aesthetic quality, realism, and adherence to the prompt is used to refine their output.

Deployment on resource-constrained devices

Model compression techniques are fundamental to the field of Edge AI and edge computing:

  • Mobile Devices: Tech companies use pruning and quantization to deploy large models on smartphones for applications like real-time language translation

  • Frameworks like TensorFlow Lite: Provide tools specifically for post-training quantization, enabling deployment on devices with limited memory and processing power

Tools and frameworks

Open-source libraries

FrameworkOrganizationKey Features
TRLHugging FaceSFT, DPO, PPO, GRPO trainers
PEFTHugging FaceLoRA, QLoRA, 60-80% memory reduction
AxolotlOpen-sourceYAML config, multi-method support
UnslothOpen-sourceClaims 2x faster training with 70% less VRAM
TorchtunePyTorchPyTorch-native library; no longer actively maintained (development wound down in 2025)

Expanded article table

TRL (Transformer Reinforcement Learning) serves as a comprehensive full-stack library for post-training foundation models, supporting SFT, GRPO, DPO, Reward, and PPO trainers.[26]

Commercial platforms

  • OpenAI Fine-Tuning API: As of 2024, supported GPT-4o and GPT-3.5 Turbo

  • Amazon Bedrock: Managed fine-tuning for Claude, Llama, Titan

  • Microsoft Foundry (formerly Azure AI Foundry): DPO and SFT for multiple models

Challenges and limitations

Despite its transformative impact, the post-training phase faces significant technical, ethical, and practical challenges:

Technical challenges

  • Catastrophic forgetting: When a pre-trained model is fine-tuned on a new, narrow dataset, it risks "forgetting" some of the general knowledge it learned during pre-training

  • Reward hacking and over-optimization: In RLHF, the model's objective is to maximize the score from the reward model, which can lead to exploits where the model achieves high rewards without actually fulfilling user intent

  • Accuracy degradation: Model compression techniques often involve a trade-off between efficiency and accuracy

  • Alignment problem: Difficulty translating abstract, nuanced human values into concrete reward functions or preference pairs

Data dependency and quality

Post-training alignment methods are highly sensitive to the data they are trained on. The quality and diversity of human-written demonstrations for SFT and the preference labels for RLHF/DPO directly determine the quality and biases of the final model. If preference data is collected from a non-diverse group or contains inherent biases, the aligned model will learn and amplify those biases.

Vulnerability to adversarial attacks

The alignment process can create new vulnerabilities. Adversarial attacks, or "jailbreaking", involve crafting specific prompts designed to bypass a model's safety guardrails and elicit forbidden or harmful behavior. Research has shown that if a negative behavior is suppressed but not eliminated from the model's capabilities, carefully designed prompts can re-activate it.

The waterbed effect

These challenges can be conceptualized through a "waterbed effect." Post-training applies strong optimization pressure to improve one aspect of the model's behavior (for example reducing toxicity). However, this pressure can cause unintended consequences in other areas, much like pushing down on one part of a waterbed causes another part to bulge up. For example, making a model overly cautious to avoid harm might severely reduce its helpfulness and creativity.

Resource requirements

AspectRequirement
GPU Memory14-80GB depending on model size
Training TimeHours to days
Data Annotation$10K-$200K for preference data
Engineering ExpertiseML engineers for tuning and deployment

Expanded article table

Recent advances

Novel techniques (2023-2025)

Reinforcement Learning with Verifiable Rewards (RLVR) uses objective signals like code execution results or theorem proofs as rewards, driving dramatic reasoning improvements in OpenAI o1 and DeepSeek-R1.[4][29] In RLVR, the model is rewarded for reaching correct answers verified by a deterministic checker (a math solution checker, a unit test, a compiler) rather than by a learned reward model, which sharply reduces reward hacking on tasks with a definite ground truth.[30]

Test-time scaling allocates more compute during inference for harder problems through chain-of-thought prompting, self-consistency, and iterative refinement.[27] OpenAI observed that o1's performance "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)," establishing post-training and inference-time compute as a new scaling axis alongside pre-training.[31]

Efficiency improvements

  • Memory optimization: Liger Kernel delivers 20% throughput increase, 60% memory reduction

  • Synthetic data generation: Teacher models create training data at scale

  • Distributed training: FSDP, DeepSpeed ZeRO enable multi-node training

Future directions

Critical research problems

  • Data efficiency: Achieving strong performance with 100-1,000 examples

  • Catastrophic forgetting: Maintaining general capabilities during specialization

  • Reward specification: Encoding complex human values

  • Scalability: Post-training 100B+ parameter models efficiently

  • Continual learning: Models that can dynamically adapt to new information and evolving user preferences over time

  • AI feedback (RLAIF): Replacing human annotations with AI-generated feedback

  • Personalization: Models adapting to individual users over time

  • Multimodal integration: Single models handling text, images, video, audio

  • Test-time optimization: Dynamic compute allocation based on query difficulty

  • Data-free compression: Quantization and pruning methods that don't require calibration datasets

Market impact

The enterprise LLM API market doubled in six months to reach $8.4 billion in the first half of 2025, up from $3.5 billion in late 2024, according to Menlo Ventures' 2025 Mid-Year LLM Market Update; the same report found Anthropic had overtaken OpenAI as the leading enterprise model API with 32% of usage.[28] Post-training is a primary driver of this growth, enabling enterprise customization and alignment.

Post-training is shifting from being an afterthought to being the central stage for AI innovation. The ability to effectively and efficiently refine, align, and optimize foundation models is becoming the key competitive differentiator in the AI industry, defining who can build the most capable, reliable, and practical AI applications. As the industry matures, post-training represents the "product finishing" phase that transforms powerful but generic AI engines into tailored, polished, and deployable solutions.

Continue reading

More from See also

Topics:Artificial IntelligenceDeep LearningLarge Language ModelsMachine LearningNatural Language Processing

References

  1. ^1 ^2 ^3 ^4Testuggine, D. "A Primer on LLM Post-Training." PyTorch Blog, August 26, 2025. pytorch.org/...a-primer-on-llm-post-training
  2. ^DeepLearning.AI. "Post-training of LLMs." Short course overview. deeplearning.ai/...post-training-of-llms
  3. ^1 ^2 ^3Grattafiori, A. et al. "The Llama 3 Herd of Models." Meta AI, 2024. arXiv:2407.21783. arxiv.org/...2407.21783
  4. ^1 ^2 ^3 ^4Raschka, S. "The State of LLMs 2025: Reasoning Models and Reinforcement Learning." Ahead of AI. magazine.sebastianraschka.com/...state-of-llms-2025
  5. ^1 ^2Hugging Face. "Quantization." Optimum and Transformers documentation. huggingface.co/...quantization
  6. ^Kumar, K. et al. "LLM Post-Training: A Deep Dive into Reasoning Large Language Models." 2025. arXiv:2502.21321. arxiv.org/...2502.21321
  7. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Ouyang, L. et al. "Training language models to follow instructions with human feedback" (InstructGPT). OpenAI, March 2022. arXiv:2203.02155. arxiv.org/...2203.02155
  8. ^Lambert, N. et al. "Illustrating Reinforcement Learning from Human Feedback (RLHF)." Hugging Face Blog, 2022. huggingface.co/...rlhf
  9. ^NVIDIA. "How Scaling Laws Drive Smarter, More Powerful AI." NVIDIA Blog, February 12, 2025. blogs.nvidia.com/...ai-scaling-laws
  10. ^1 ^2 ^3 ^4 ^5Rafailov, R. et al. "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model." Stanford University, May 2023. arXiv:2305.18290. arxiv.org/...2305.18290
  11. ^1 ^2 ^3Bai, Y. et al. "Constitutional AI: Harmlessness from AI Feedback." Anthropic, December 2022. arXiv:2212.08073. arxiv.org/...2212.08073
  12. ^Anthropic. "Claude's Constitution." Anthropic, 2023. anthropic.com/...claudes-constitution
  13. ^Ethayarajh, K. et al. "KTO: Model Alignment as Prospect Theoretic Optimization." 2024. arXiv:2402.01306. arxiv.org/...2402.01306
  14. ^Hong, J. et al. "ORPO: Monolithic Preference Optimization without Reference Model." 2024. arXiv:2403.07691. arxiv.org/...2403.07691
  15. ^Shao, Z. et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (introduces GRPO). DeepSeek, 2024. arXiv:2402.03300. arxiv.org/...2402.03300
  16. ^Han, S. et al. "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding." ICLR 2016. arXiv:1510.00149. arxiv.org/...1510.00149
  17. ^Lin, J. et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. arXiv:2306.00978. arxiv.org/...2306.00978
  18. ^Frantar, E. and Alistarh, D. "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot." ICML 2023. arXiv:2301.00774. arxiv.org/...2301.00774
  19. ^Hinton, G., Vinyals, O., Dean, J. "Distilling the Knowledge in a Neural Network." 2015. arXiv:1503.02531. arxiv.org/...1503.02531
  20. ^Christiano, P. et al. "Deep Reinforcement Learning from Human Preferences." OpenAI and DeepMind, June 2017. arXiv:1706.03741. arxiv.org/...1706.03741
  21. ^Wei, J. et al. "Finetuned Language Models Are Zero-Shot Learners" (FLAN). Google, 2021. arXiv:2109.01652. arxiv.org/...2109.01652
  22. ^Malik, A. "ChatGPT reaches 900M weekly active users." TechCrunch, February 27, 2026. techcrunch.com/...reaches-900m-weekly-active-users
  23. ^Google DeepMind. "Gemini: A Family of Highly Capable Multimodal Models." 2023. arXiv:2312.11805. arxiv.org/...2312.11805
  24. ^Lambert, N. "The RLHF Book." rlhfbook.com
  25. ^Google. "MedGemma: Medical Multimodal Models." Google Health AI, 2025. developers.google.com/...medgemma
  26. ^Hugging Face. "TRL: Transformer Reinforcement Learning." Documentation. huggingface.co/...trl
  27. ^Snell, C. et al. "Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters." 2024. arXiv:2408.03314. arxiv.org/...2408.03314
  28. ^Menlo Ventures. "2025 Mid-Year LLM Market Update." July 2025. menlovc.com/...2025-mid-year-llm-market-update
  29. ^1 ^2 ^3 ^4 ^5DeepSeek-AI. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." January 2025. arXiv:2501.12948. arxiv.org/...2501.12948
  30. ^Lambert, N. et al. "Tulu 3: Pushing Frontiers in Open Language Model Post-Training" (RLVR). Allen Institute for AI, 2024. arXiv:2411.15124. arxiv.org/...2411.15124
  31. ^1 ^2OpenAI. "Learning to Reason with LLMs" (o1). OpenAI, September 12, 2024. openai.com/...learning-to-reason-with-llms
  32. ^AI Wiki. "Post-training: an RLHF example". CC0 1.0. Original geometric illustration; not a reproduction of a source figure. Technical basis: Ouyang et al., Training language models to follow instructions with human feedback.
  33. ^Anthropic. "Claude's new constitution." Anthropic, January 2026. anthropic.com/...claude-new-constitution

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

10 revisions · v10 · 4,913 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check of the full article against its cited sources (110 claims, 30 sources fetched); 7 material + 22 minor fixed

Cite this page: AI Wiki. "Post-training." aiwiki.ai, updated 28 Oct 2026, fact-checked 2 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/post-training

Edit article