Together Post-Training

Together Post-Training is a managed service from Together AI for customizing open-weight large language models with reinforcement learning (RL) and supervised fine-tuning (SFT). The customer writes the training loop in Python, including the data, reward functions, environments and loss, and calls the service at each step; Together AI runs the trainer, the rollout (sampling) engines and the weight synchronization between them on dedicated NVIDIA B200 GPUs.^[1][3][4] Together AI opened the service to all customers on 19 October 2026, after a beta that, according to an early customer, began in April.^[1][2] The service bills per GPU-minute rather than per token, offers LoRA and (for some models) full-weight training, and includes a client that runs scripts written for Tinker, the training API from Thinking Machines Lab, after a one-line import change.^[1][8][11]

Overview

Together AI already sold a job-based fine-tuning service, in which the customer uploads a dataset, submits a job and lets Together run the whole loop. Post-Training is the lower-level alternative. Its documentation describes it as "more powerful but more complex": the customer's code drives the loop and the remote GPUs run the heavy steps (forward and backward passes, optimizer steps and sampling).^[3] The table below summarizes the differences as Together's documentation lists them.^[3]

The training loop can run on any machine with the Together SDK installed, including a laptop with no GPU.^[4] In Together's launch post, the company argued that many organizations are now training specialist models to match frontier-model quality at lower cost, and that RL infrastructure has become complex enough that companies "never start it, or start and stall".^[1]

Launch

Together AI published the launch post on its blog and as an X article on 19 October 2026, under the title "Together Post-Training: frontier-grade model shaping, the way that fits you". The blog version credits 26 authors.^[1][2] Unite.AI and TokenPost reported the launch the same day, and Data Phoenix followed on 20 October.^[19][20][21]

The service existed in earlier form. An archived copy of Together's "Custom Training" page from 5 August 2026 carried the heading "Reinforcement Learning - now in beta", described "RL and SFT, full-weight and LoRA, on frontier open models", and asked visitors to request access.^[22] In the launch post, Adit Jain of Collinear AI said his team had worked with Together "since the launch of its Post-Training beta in April".^[1] Data Phoenix, by contrast, reported that Together had introduced the underlying custom-training system in a limited closed beta on 23 July.^[21]

The launch post quoted three early-access customers:^[1]

Together also offers forward-deployed engineers who work with customers on model selection, data, training environments, evaluation and deployment.^[1]

Architecture and concepts

A run is split between the customer's code and a model resource on Together's GPUs. The model resource holds the base model and does the heavy computation; the customer's code loads data, builds batches, scores outputs and decides when to stop.^[4]

One RL step in the documentation's description has four service calls around the customer's own scoring: sample_batch generates completions, the customer's code scores them, forward_backward computes the loss and gradients, optim_step updates the weights, and weights_sync publishes the new weights to the generator replicas so the next step samples from the updated policy. SFT and DPO skip sampling and weight sync.^[3][4] The trainer and the generators hold separate copies of the weights, and the generators' copy only changes on weights_sync.^[4]

Because the base weights are frozen in LoRA mode, several LoRA sessions can share one resource, each with its own data, hyperparameters and checkpoints, at the cost of a single resource. A full-weight resource hosts a single session.^[4][6] Sessions are free; only the resource is billed.^[8] A resource that goes one hour without an operation stops automatically and its sessions expire.^[14]

Optimizer choice is Adam by default. The Muon optimizer is available only for full-weight training.^[6][14]

Supported methods and losses

The documentation lists three training methods: SFT on example responses, direct preference optimization (DPO) on preference pairs, and GRPO against a reward function the customer writes. GRPO can also train tool-using agents over multi-turn episodes.^[6] For SFT and GRPO the service runs a built-in loss; DPO is implemented as a custom loss in the customer's process.^[6]

The built-in losses, as listed in the GRPO documentation, are:^[7]

GRPO was introduced in DeepSeek's DeepSeekMath paper as a variant of proximal policy optimization,^[18] and CISPO was proposed in the MiniMax-M1 paper, which describes it as clipping importance sampling weights rather than token updates.^[17]

For objectives the built-in losses do not cover, the customer can compute a custom loss locally in PyTorch. The service returns per-token log-probabilities from a forward call, the customer's code computes the loss and its gradient with respect to those log-probabilities, and custom_forward_backward sends the gradients back to the trainer, which backpropagates them through the model. Because the forward pass runs twice, a custom-loss step takes longer than a built-in one. A custom loss sees only the log-probability of each target token, so objectives that need the full vocabulary distribution (such as an exact entropy bonus) cannot be expressed this way.^[24]

Synchronous and asynchronous RL

The default is lockstep, in which rollouts finish before training, leaving the generators idle while the trainer works and vice versa. Asynchronous mode overlaps rollout collection with optimizer steps. A max_steps_off_policy setting bounds how many optimizer steps old a rollout group may be, and older groups are dropped as stale. Three weight-sync modes exist: synchronous, background publish, and pipeline; pipeline mode requires full-weight training.^[7] Together's product page says async samples are "one step behind, and the loss corrects for the gap", and recommends starting with sync.^[10]

Supported models

The launch post named Muse Spark Open, GLM 5.3, Kimi K3 and Qwen 3.8-27B as launch models and said more would follow.^[1] The documentation catalog, which syncs daily from the service's supported-models endpoint, listed seven B200 configurations as of 30 October 2026:^[5]

The documentation says the service runs only on B200 GPUs for now, with support for B300 and GB300 GPUs to "be added soon", and supports text models only.^[5] Together's own materials did not agree on every detail: the product page labels Qwen3.6 35B A3B as supporting full-weight training, while the documentation catalog lists it as LoRA only, a mismatch Data Phoenix also noted.^[5][10][21]

Training-inference alignment

In RL for language models, the policy that generates rollouts (an inference engine) and the policy being trained (a training engine) are supposed to be the same model, but differences in numerics between the two stacks can make the same weights produce slightly different probabilities. Together's launch post says such mismatches can make a run "underperform or even collapse", and that the service removes "numerous sources of policy drift" using standard techniques including Rollout Routing Replay as well as its own optimizations.^[1]

Rollout Routing Replay (R3) comes from the 2025 paper "Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers" by Wenhan Ma and six co-authors, including Luo Fuli. The paper reports that in mixture-of-experts models the router can pick different experts at inference and training time, and even across repeated forward passes. R3 records the routing distributions from the inference engine and replays them during training; the authors report that it reduces the training-inference KL divergence and stabilizes RL training.^[16] Together's product page describes the same idea in its own terms: the trainer matches the generator's output precision, replays its expert choices on supported MoE models, and corrects for samples from older model versions in every RL loss.^[10]

A related problem appears at deployment. Dedicated inference serves GLM-5.3 and Kimi K3 in 4-bit formats, while training does not run on those weights, so an adapter does not exactly match the deployed model. Together's FP4 preparation job merges the adapter into the BF16 weights first, then calibrates and quantizes the merged model, which the documentation says takes about 6 hours for GLM-5.3 and about 18 hours for Kimi K3.^[13] See FP4 for background on the format.

Deployment after training

Each LoRA inference checkpoint is registered as a model in the customer's project. Several adapters can be attached to one multi-LoRA deployment of their base model on Together's dedicated inference, with each request choosing an adapter by name; attaching an adapter to a running deployment "usually takes under a minute", against 20 minutes or more to provision a new deployment, according to the documentation. The winning adapter can then be served on its own deployment, merged into the base model.^[12] Trained weights can also be downloaded as a standard Hugging Face model directory and served elsewhere.^[4]

Tinker compatibility

The Together Python SDK includes a client compatible with Tinker, which Thinking Machines Lab describes as "a cloud training API" in which users "write training loops locally" while Tinker "handles the GPUs".^[23] Replacing import tinker with import together.post_training.tinker as tinker lets an existing Tinker script run on Together, with the base model changed if needed because the two services host different model catalogs.^[1][11] The documentation lists behavioural differences, among them that sampling clients are not snapshots of earlier weights, that policy-gradient losses accept only 0 or 1 weights, that loss inputs must be one-dimensional (so top-K distillation is unsupported), that the maximum LoRA rank is 64 on Qwen models and 32 on others, and that the Tinker client works only in LoRA mode.^[6][11]

The open-source post-training cookbook on GitHub contains recipes for GRPO on GSM8K, DeepMath and Game of 24, a multi-turn tool-use environment, SFT on No Robots, DPO on HelpSteer3 as a custom-loss demo, and multi-LoRA hyperparameter sweeps.^[15]

Availability and pricing

Post-Training bills each model resource per GPU-minute while it is in the ready state; queueing, startup and shutdown are not billed. The rate is the per-GPU-hour price multiplied by the number of GPUs, which depends on the model, the training mode and the number of generator replicas.^[8] Together's pricing page listed the HGX B200 at $11.00 per GPU-hour, with a promotional $9.00 rate "valid until 12/14/2026".^[9] The blog post describes the promotion as lasting "the first eight weeks"; the X article version of the same post said "six weeks", and Unite.AI repeated the six-week figure.^[1][2][19]

The documentation gives a worked example: Qwen3.8-27B with LoRA and one generator replica on B200 uses four trainer GPUs and four generator GPUs, for $88 per hour at $11 per GPU-hour, or half that with no generator replica.^[8] Creating a resource requires a credit balance of at least $100. If a ready resource fails, Together refunds the last 3 hours of runtime for a LoRA resource or the last 6 hours for a full-weight resource.^[8]

Together states that per-GPU billing becomes cheaper than token-based pricing "as soon as you scale up the experiments".^[1] Data Phoenix noted that the official launch materials did not include an independent benchmark for Together's cost and speed claims.^[21]

Context

Together Post-Training exposes low-level training primitives (forward-backward, optimizer step, sampling, weight sync) instead of a fixed fine-tuning job. Tinker, from Thinking Machines Lab, uses the same split between a training loop written locally and GPUs run by the provider, and Together ships a Tinker-compatible client alongside its own SDK.^[3][11][23] Within Together AI, the service complements the job-based fine-tuning product, and its checkpoints deploy directly to Together's dedicated inference.^[3][12] See post-training and fine-tuning for the general techniques.

Continue reading

• LLoRA (Low-Rank Adaptation)Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model.Mentioned 24 times above
• TTinker (Thinking Machines Lab)Tinker is a fine-tuning API for open-weight language models, built by Thinking Machines Lab, the artificial intelligence startup founded by former OpenAI chief technology officer Mira Murati.Mentioned 11 times above
• Together AITogether AI is an AI cloud platform for running, fine-tuning, and training open-source foundation models at high performance and low cost.Mentioned 7 times above
• Reinforcement learningReinforcement learning (RL) is a branch of machine learning in which an agent learns to make decisions by taking actions in an environment to maximize a cumulative reward signal, discovering good behavior…Mentioned 4 times above
• Thinking Machines LabThinking Machines Lab is an American artificial intelligence research and product company, headquartered in San Francisco, California, that was founded in February 2025 by Mira Murati, the former chief…Mentioned 3 times above
• Large Language ModelA large language model (LLM) is an artificial intelligence system built on a transformer neural network with billions to trillions of parameters, trained on massive text corpora to predict the next token in a…Linked in the introduction

More to read

• GRPO
• Fine-tuning
• DPO
• FP4 (4-bit floating point)
• Proximal Policy Optimization (PPO)
• Luo Fuli

Topics:AI InfrastructureDeveloper ToolsReinforcement LearningTraining & Optimization

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Together AI. "Together Post-Training: frontier-grade model shaping, the way that fits you." Together AI blog, 19 October 2026. together.ai/...together-post-training
2. ^1 ^2 ^3Together AI (@togethercompute). X article "Together Post-Training: frontier-grade model shaping, the way that fits you." 19 October 2026. x.com/...2112229246802301248
3. ^1 ^2 ^3 ^4 ^5 ^6Together AI. "Overview." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...overview
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Together AI. "Concepts." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...concepts
5. ^1 ^2 ^3Together AI. "Supported models." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...supported-models
6. ^1 ^2 ^3 ^4 ^5Together AI. "Training methods overview." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...training-methods
7. ^1 ^2Together AI. "Reinforcement learning with GRPO." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...grpo
8. ^1 ^2 ^3 ^4 ^5Together AI. "Pricing." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...pricing
9. ^Together AI. "Pricing." Accessed 30 October 2026. together.ai/pricing
10. ^1 ^2 ^3Together AI. "Together Post-Training: RL and SFT for Open Models." Product page. Accessed 30 October 2026. together.ai/post-training
11. ^1 ^2 ^3 ^4Together AI. "Migrate from Tinker." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...migrate-from-tinker
12. ^1 ^2Together AI. "Multi-LoRA deployment." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...multi-lora-deployment
13. ^Together AI. "Prepare for FP4 inference." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...prepare-for-fp4-inference
14. ^1 ^2 ^3Together AI. "Manage model resources." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...model-resources
15. ^Together AI. "post-training-cookbook." GitHub repository. Accessed 30 October 2026. github.com/...post-training-cookbook
16. ^Ma, Wenhan; Zhang, Hailin; Zhao, Liang; Song, Yifan; Wang, Yudong; Sui, Zhifang; Luo, Fuli. "Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers." arXiv:2510.11370, 2025. arxiv.org/...2510.11370
17. ^MiniMax. "MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention." arXiv:2506.13585, 2025. arxiv.org/...2506.13585
18. ^Shao, Zhihong et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300, 2024. arxiv.org/...2402.03300
19. ^1 ^2Unite.AI. "Together AI Launches Together Post-Training Service for Custom Models." October 2026. unite.ai/...ost-training-service-for-custom-models
20. ^Yoon, Simon. "Together AI Launches Post-Training Service With Reserved GPU Capacity." TokenPost, 19 October 2026. tokenpost.com/...35231
21. ^1 ^2 ^3 ^4Data Phoenix. "Together AI launches managed post-training service for open models." 20 October 2026. dataphoenix.info/...together-ai-post-training-launch
22. ^Together AI. "Custom Training" page (archived 5 August 2026). Internet Archive. web.archive.org/...custom-training
23. ^1 ^2Thinking Machines Lab. "Tinker Documentation." Accessed 30 October 2026. tinker-docs.thinkingmachines.ai
24. ^Together AI. "Write a custom loss function." Post-training documentation. Accessed 30 October 2026. docs.together.ai/...custom-loss-functions

Show all 24 references