SkyRL

SkyRL is an open-source reinforcement learning (RL) library for post-training large language models and multi-turn agents. It is developed by NovaSky, a student-and-advisor group at the Sky Computing Lab of the University of California, Berkeley, in collaboration with Anyscale.^[1][4] The code is published at github.com/NovaSky-AI/SkyRL under the Apache License 2.0; the repository was created on April 22, 2025, and as of October 30, 2026 the GitHub API reported 2,476 stars, 472 forks and 115 contributors.^[1][2] SkyRL began in May 2025 as a pipeline for training multi-turn, tool-using agents on long-horizon tasks such as software-engineering benchmarks, was rebuilt in mid-2025 as a modular framework, and has since become a single skyrl package with FSDP, Megatron and JAX training backends, vLLM for generation, and an implementation of the Tinker training API.^[1][3][10]

In October 2026 SkyRL gained "day-0" RL recipes for newly released open-weight models: one for Beam from Reflection AI, published by the SkyRL team on October 16, and one for Mistral Large 4, which Mistral and the vLLM team announced on October 29, the day Mistral AI released that model's weights.^{[18][19][20][39]} The Mistral Large 4 work was announced on the vLLM blog and on X, but on October 30, 2026 its code was still in five open, unmerged pull requests.^[19][20][23]

Background

RL post-training of a language model alternates between two jobs. An inference engine generates responses (rollouts) and records the probability of each sampled token; a training engine then scores those responses with a reward and updates the weights, which are copied back to the inference engine for the next round. The SkyRL-v0 announcement argued that most open RL frameworks at the time were tuned for short, stateless interactions such as search-augmented reasoning or single code executions, while tasks like SWE-bench need long-horizon planning in stateful environments, which makes both the infrastructure and the training harder.^[5]

A second problem became central to SkyRL's later work: the trainer and the inference engine compute slightly different probabilities for the same token because they use different kernels, batch shapes, parallel layouts and numeric formats. SkyRL's documentation lists kernel mismatch, inconsistent mixture-of-experts routing and different parallelism settings as sources of this drift, and provides corrections for it.^[13]

Developers

NovaSky describes itself as "a collaborative initiative led by students and advisors at UC Berkeley's Sky Computing Lab". Its About page lists Dacheng Li, Shiyi Cao, Shu Liu and Tyler Griggs as student leads and Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez and Ion Stoica as advisors.^[4] The SkyRL README says the work is done at the Berkeley Sky Computing Lab in collaboration with Anyscale, with compute support from Anyscale, Databricks, NVIDIA, Lambda, AMD, AWS, Modal and Daytona.^[1] The SkyRL-v0 author list combined UC Berkeley, Anyscale and All Hands AI researchers, and an October 2026 SkyRL blog post refers to "the SkyRL team from Anyscale".^[5][30]

Before SkyRL, NovaSky released Sky-T1-32B-Preview (January 10, 2025), a reasoning model the group said it trained for under $450. Sky-T1 came from a separate repository, SkyThought, and was trained by supervised fine-tuning of Qwen2.5-32B-Instruct on 17,000 traces using Llama-Factory, not with SkyRL, which did not yet exist.^[28] A follow-up 7B model, Sky-T1-7B, added RL stages using the PRIME and RLOO algorithms.^[37]

Components

The README lists the following parts of the repository as of October 30, 2026.^[1]

The repository also contains skycap, a separate package for capturing agent trajectories: an agent harness points its unchanged OpenAI-compatible client at a per-trajectory URL, and skycap records each model call and returns training samples.^[38]

The repository's LICENSE file is Apache 2.0, while the skyrl-gym and skyrl-train packages declare the MIT license in their PyPI metadata.^[1][33]

Architecture

SkyRL's system overview splits training into a Trainer, which runs the optimization step for the configured algorithm, and a Generator, which produces complete trajectories and their rewards. The Generator in turn contains an InferenceEngine (model completions) and an Environment (task logic, observations and rewards), and a Controller handles placement, initialization and control flow. Training workers run as Ray actors.^[8]

Version 0.2.0 of the unified package (April 2026) moved inference onto HTTP, and v0.3.0 removed the older code path. A RemoteInferenceClient sends routed generation requests through a VLLMRouter (a wrapper around vLLM's router) to standard vLLM API servers, and sends control requests (pause, resume, sleep, wake and weight sync) to every server directly.^[3][9] Requests from one trajectory carry a session ID so that every turn lands on the same replica and reuses its prefix cache. A sticky_least_loaded policy, added in v0.3.0, sends new trajectories to the least-busy replica; SkyRL reported up to 1.4 times higher throughput on a multi-turn agentic task with GLM 4.7.^[3][9] SkyRL uses vLLM's native weight-syncing and pause/resume APIs, which SkyRL and vLLM developers described together in May 2026.^[31]

Version 0.4.0 (October 2026) added several weight-transfer paths: delta weight sync, which sends compressed XOR differences against the previous BF16 weights over disk, S3 or GCS-style storage for disaggregated clusters; sharded transfer over NIXL via Ray Direct Transport, which SkyRL says moved the BF16 weights of Kimi K2 in 7.53 seconds across 48 nodes of eight H100 GPUs; and in-memory LoRA adapter sync over NCCL or CUDA IPC instead of through disk.^[17] The framework supports colocated training, where trainer and inference share GPUs, as well as one-step off-policy and fully asynchronous training; fully asynchronous training arrived in skyrl_train v0.3.0 in December 2025.^[1][3]

Backends and hardware

The FSDP backend accepts any model supported by Hugging Face Transformers and vLLM; the docs describe it as suited to dense models up to 32B parameters. The Megatron backend is meant for large MoE models above 100B parameters.^[10] On October 30, 2026 the docs listed full Megatron support for Qwen3.5 (dense and MoE up to 397B-A17B), Nemotron 3, GLM-4.7-Flash and GLM-4.7, GLM-5.3-Flash, Qwen3 (up to 235B-A22B) and Moonlight-16B-A3B, and JAX support for dense Qwen3.5, DeepSeek-V3, Llama 3 and Qwen3.^[10] The v0.4.0 notes added GLM-5.3, Qwen3.8 and Nemotron 3.5 Lightning, plus experimental LoRA training of Kimi K2.6 and K2.7, which SkyRL says fits a 1-trillion-parameter model on two B300 nodes by serving in INT4 and fake-quantizing the trainer.^[17] Commits merged on October 30, 2026 added Megatron support for the text model of DeepSeek V4.1-Flash and for a model the commit calls Qwen3.8-Flash-Next.^[36]

Algorithms

SkyRL implements PPO-style training with interchangeable advantage estimators and policy losses, and a registry for adding custom ones without changing the core code.^[14] The source code on October 30, 2026 defined these built-in options:^[15]

SkyRL supports the DAPO recipe (asymmetric "clip-higher" bounds, dynamic sampling that discards prompts whose samples all get the same reward, token-level loss averaging and overlong-response penalties) mostly through configuration; the soft overlong penalty is applied by a custom trainer class in SkyRL's DAPO example.^[16] For trainer-inference mismatch, SkyRL offers truncated importance sampling (TIS) at token or sequence level, sequence masking by the geometric mean of importance ratios, token masking, rollout routing replay (R3) for MoE models, top-p and top-k sampler replay (v0.4.0) and score centering.^[13][17] SkyRL's own comparison on a 1.5B distilled model found geometric sequence masking gave better evaluation results than TIS in one-step off-policy training.^[13]

Tinker API

SkyRL implements the Tinker API that Thinking Machines Lab introduced, a small set of calls (forward_backward, forward, optim_step, sample, save_checkpoint and load_checkpoint). SkyRL says any script written against it runs on local hardware with "zero code changes", on FSDP or Megatron for training and vLLM for sampling.^[12] The work started as SkyRL tx in October 2025, and skyrl_train v0.4.0 (February 13, 2026) brought full Tinker API support to the main training stack.^[1][3] The docs list multi-tenant LoRA training and sampling as supported on Megatron with vLLM, but not yet on FSDP.^[12]

Projects and models

The checkpoints released with SkyRL-v0, SkyRL-SQL and SkyRL-Agent are hosted on the NovaSky-AI Hugging Face organization, while Mercor published its recipe under its own GitHub and Hugging Face accounts.^[25][29] Outside groups have also built on the library: the CodeScout paper (March 2026) used SkyRL as its training backend for code-search agents, and the Endless Terminals paper (January 2026) trained terminal agents with PPO and states that "our implementation uses SkyRL".^[26][27] The README lists further projects that use SkyRL, including the biomedical reasoning agent Biomni-R0 and OpenThoughts-Agent.^[1] The project also supports environments from Harbor (from the team behind Terminal-Bench) and Prime Intellect's Environments Hub.^[1][3][25]

In August 2026 Alexander Jiang and the SkyRL team described IsoExec, an opt-in execution contract with batch-invariant kernels that made vLLM and Megatron compute nearly identical log-probabilities in one 50-step test, at a reported 25% step-time cost; the vLLM article covers it in detail.^[32]

Release history

Day-zero support for Beam (October 2026)

On October 16, 2026, the day Reflection AI released Beam's weights, SkyRL published LoRA, BF16 full fine-tuning and MXFP8 RL recipes for the 501B-parameter MoE model, trained with Megatron and served with vLLM. SkyRL called it "the first Day-0 RL support effort in SkyRL". On the DAPO recipe it reported AIME 2025 avg@8 accuracy rising from 61% to 79% in 50 training steps, and said the changes it made let Beam be trained with LoRA on as few as two B200 nodes.^[18] The code lives on a separate beam-release branch.^[1] The Beam article covers the runs and the engineering changes in detail.

Day-zero support for Mistral Large 4 (October 2026)

Mistral AI released the weights of Mistral Large 4 (a 1-trillion-parameter MoE model with 128 routed experts, four active per token) on October 29, 2026.^[19][39] The same day, the vLLM blog post "Day 0 support for Mistral Large 4 on vLLM", credited to "Mistral and the vLLM Team", included a section saying the teams had "enabled Day 0 RL for Mistral Large 4 in SkyRL".^[19] A GitHub issue opened on the SkyRL repository on October 29, titled "[RFC] Day-0 RL support for Mistral Large 4 (NVFP4) on FSDP and Megatron", tracks the work and lists five stacked pull requests (#2576 to #2580).^[20] The vLLM project's X account promoted the results on October 30.^[23] As of October 30, 2026, the issue and all five pull requests were open, and the pull requests, all marked as drafts, had been submitted from a contributor's fork rather than merged into SkyRL's main branch; they were still open and unmerged on October 31.^[20][21]

Recipe

The run fine-tunes LoRA adapters on top of Mistral's NVFP4 checkpoint, with vLLM generating rollouts and either FSDP2 or Megatron training the adapters, on eight NVIDIA GB200 nodes.^[19] The example configuration in pull request #2576 describes that as eight nodes of four GPUs, or 32 GPUs in total, with training and inference colocated on the same GPUs.^[21]

The terms in the recipe mean the following:

• LoRA (rank 32). Low-rank adaptation freezes the original weights and trains small added matrices; the rank sets their size. Only these adapters are trained and copied to vLLM after each step.^[21]
• NVFP4 quantization-aware training. NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs. In quantization-aware training, the trainer reproduces the rounding the inference engine applies, so both compute the same numbers. According to the Megatron pull request, the expert weights stay in their NVFP4 form in memory, and the trainer applies NVFP4 and FP8 activation quantization that matches the rollout engine.^[22]
• FP8 KV-cache simulation. Mistral Large 4 uses multi-head latent attention, which stores a compressed "latent" version of each token's keys and values in the KV cache, and vLLM serves the model with that cache in FP8 by default. The trainer simulates the FP8 cache contents so its attention matches what vLLM computed during the rollout.^[19][21]
• Routing replay (R3). In a mixture-of-experts layer a router picks which experts process each token. Small numeric differences can make the trainer pick different experts from those vLLM used, which changes the token probabilities. Rollout Routing Replay records vLLM's expert choices and forces the trainer to reuse them. It was introduced in October 2025 by researchers from Peking University and Xiaomi, who reported that it reduced training-inference divergence and prevented RL collapse in their MoE experiments.^[13][24]
• Token-level truncated importance sampling. Because the tokens were sampled by vLLM, the trainer reweights each token's loss by the ratio of its own probability to vLLM's probability, capped at a maximum (2.0 here) so that a few extreme ratios cannot dominate the update. SkyRL's docs attribute the technique to a 2025 analysis by Yao and colleagues.^[13][21]
• Logprob gap. The mean absolute difference between the log-probabilities that the trainer and vLLM assign to the same sampled tokens, used as a measure of how far apart the two systems are.^[13]

Reported results

All figures below are self-reported by the teams involved and have not been independently reproduced.

The vLLM post describes the accuracy change as about 64% to about 75% "within 75 steps", and the X post says "in 75 training steps".^[19][23] The tracking issue shows that the run continued past that point and that accuracy was slightly lower, 0.72 to 0.73, at steps 95-100.^[20] The two sources also give different speed comparisons for the Megatron backend: the issue's step times correspond to a step roughly 11% shorter than FSDP's, measured "after the sync/offload/balancing work in this series", while the vLLM post says about 30%. Neither source explains the difference.^[19][20]

Continue reading

• vLLMvLLM is an open-source inference and serving engine for large language models. It is software for running already trained models, either through a Python interface for offline work or through a network server…Mentioned 37 times above
• MMegatron-LMMegatron-LM is NVIDIA's open-source framework for training very large transformer language models across GPU clustersMentioned 19 times above
• TTinker (Thinking Machines Lab)Tinker is a fine-tuning API for open-weight language models, built by Thinking Machines Lab, the artificial intelligence startup founded by former OpenAI chief technology officer Mira Murati.Mentioned 11 times above
• FFully Sharded Data Parallel (FSDP)Fully Sharded Data Parallel (FSDP) is a distributed training technique implemented in PyTorch that shards a model's parameters, gradients, and optimizer states across data-parallel workers, allowing models…Mentioned 10 times above
• AnyscaleAnyscale is an American technology company, founded in 2019 by the creators of Ray at the University of California, Berkeley, that develops the Anyscale Platform: a fully managed compute service built on Ray…Mentioned 5 times above
• JAXJAX is an open-source Python library for accelerator-oriented array computation and program transformation.Mentioned 3 times above

More to read

• Reinforcement learning
• Large Language Model
• University of California, Berkeley
• Apache License 2.0
• Mistral Large 4
• Beam (Reflection)

Topics:Developer ToolsOpen Source AIReinforcement LearningTraining & Optimization

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20NovaSky-AI. "SkyRL: A Modular Full-stack RL Library for LLMs". GitHub repository README and LICENSE. Accessed October 30, 2026.
2. ^GitHub REST API. Repository record for NovaSky-AI/SkyRL and contributors list. Accessed October 30, 2026.
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17NovaSky-AI. SkyRL releases: skyrl_train v0.1.0 to v0.4.0 and skyrl v0.1.0 to v0.4.0 release notes. GitHub. Accessed October 30, 2026.
4. ^1 ^2NovaSky. "About Us". Sky Computing Lab, UC Berkeley. Accessed October 30, 2026.
5. ^1 ^2 ^3 ^4Cao, Shiyi, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, et al. "SkyRL-v0: Train Real-World Long-Horizon Agents via Reinforcement Learning". NovaSky blog, May 2025. Accessed October 30, 2026.
6. ^1 ^2Liu, Shu, Sumanth Hegde, Shiyi Cao, Alan Zhu, et al. "SkyRL-SQL: Matching GPT-4o and o4-mini on Text2SQL with Multi-Turn RL". NovaSky blog, May 20, 2025. Accessed October 30, 2026.
7. ^Cao, Shiyi, Dacheng Li, Fangzhou Zhao, et al. "SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent". arXiv:2511.16108, November 20, 2025.
8. ^SkyRL Documentation. "SkyRL System Overview". Accessed October 30, 2026.
9. ^1 ^2SkyRL Documentation. "Inference Architecture". Accessed October 30, 2026.
10. ^1 ^2 ^3 ^4SkyRL Documentation. "Supported Models". Accessed October 30, 2026.
11. ^SkyRL Documentation. "Switching Training Backends". Accessed October 30, 2026.
12. ^1 ^2 ^3SkyRL Documentation. "Tinker API". Accessed October 30, 2026.
13. ^1 ^2 ^3 ^4 ^5 ^6SkyRL Documentation. "Off-Policy Correction". Accessed October 30, 2026.
14. ^SkyRL Documentation. "Implementing Custom Algorithms". Accessed October 30, 2026.
15. ^NovaSky-AI. `ppo_utils.py` (AdvantageEstimator and PolicyLossType definitions). SkyRL main branch. Accessed October 30, 2026.
16. ^SkyRL Documentation. "DAPO". Accessed October 30, 2026.
17. ^1 ^2 ^3 ^4 ^5 ^6SkyRL Blog. "SkyRL v0.4.0 Release". October 6, 2026. Accessed October 30, 2026.
18. ^1 ^2Hegde, Sumanth, and the SkyRL team. "Day 0 Support for Beam in SkyRL". SkyRL Blog, October 16, 2026. Accessed October 30, 2026.
19. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13Mistral and the vLLM Team. "Day 0 support for Mistral Large 4 on vLLM", section "Day 0 Support for RL with SkyRL". vLLM Blog, October 29, 2026. Accessed October 30, 2026.
20. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15NovaSky-AI/SkyRL issue #2575. ["[RFC] Day-0 RL support for Mistral Large 4 (NVFP4) on FSDP and Megatron"](github.com/...2575). GitHub, opened October 29, 2026. Accessed October 30, 2026.
21. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11NovaSky-AI/SkyRL pull request #2576. "feat(fsdp): ML4 (Mistral Large 4 NVFP4) LoRA RL on FSDP2", including `examples/ml4/ml4_dapo_lora_fsdp.yaml`. GitHub, opened October 29, 2026. Accessed October 30, 2026.
22. ^NovaSky-AI/SkyRL pull request #2577. "feat(megatron): ML4 bridge, layer spec and modules with NVFP4/FP8 QAT". GitHub, opened October 29, 2026. Accessed October 30, 2026.
23. ^1 ^2 ^3vLLM (@vllm_project). Post on day-0 RL for Mistral Large 4 with SkyRL and vLLM. X, October 30, 2026.
24. ^Ma, Wenhan, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. "Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers". arXiv:2510.11370, October 13, 2025 (v2 October 21, 2025).
25. ^1 ^2 ^3Mercor and SkyRL. "Training frontier knowledge work agents: A 397B RL training guide with SkyRL". Mercor Blog, September 1, 2026. Accessed October 30, 2026.
26. ^Sutawika, Lintang, et al. "CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents". arXiv:2603.17829, March 18, 2026.
27. ^Gandhi, Kanishk, Shivam Garg, Noah D. Goodman, and Dimitris Papailiopoulos. "Endless Terminals: Scaling RL Environments for Terminal Agents". arXiv:2601.16443, January 23, 2026.
28. ^NovaSky Team. "Sky-T1: Train your own O1 preview model within $450". NovaSky blog, January 10, 2025. Accessed October 30, 2026.
29. ^NovaSky-AI. Hugging Face organization (SkyRL-Agent-7b-v0, SkyRL-Agent-8b-v0, SkyRL-Agent-14B-v0, SkyRL-SQL-7B, SA-SWE-32B and datasets). Accessed October 30, 2026.
30. ^1 ^2Basnet, Avi, and the SkyRL Team; Keith Truongcao and the Lila Sciences AI Research Team. "Scaling Training to 1M Token Context on GLM-5.3-Flash". SkyRL Blog, October 22, 2026. Accessed October 30, 2026.
31. ^Hao, Aaron, Sumanth Hegde, Kyle Sayers, Kourosh Hakhamaneshi, and the vLLM team. "Native RL APIs in vLLM". vLLM Blog, May 28, 2026. Accessed October 30, 2026.
32. ^Jiang, Alexander, and the SkyRL Team. "IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL". vLLM Blog, August 21, 2026. Accessed October 30, 2026.
33. ^Python Package Index. skyrl-gym and skyrl-train package metadata; see also `skyrl-gym/pyproject.toml`. Accessed October 30, 2026.
34. ^SkyRL Documentation. "Training with Quantized Rollouts". Accessed October 30, 2026.
35. ^SkyRL Documentation. "SkyAgent Overview". Accessed October 30, 2026.
36. ^1 ^2NovaSky-AI. SkyRL commit history (including pull requests #2567 and #2513) and SkyRL Documentation, "AMD ROCm Tinker". Accessed October 30, 2026.
37. ^NovaSky Team. "Unlocking the Potential of Reinforcement Learning in Improving Reasoning Models". NovaSky blog, February 13, 2025. Accessed October 30, 2026.
38. ^NovaSky-AI. `skycap` README. SkyRL main branch. Accessed October 30, 2026.
39. ^1 ^2Mistral AI. "Mistral Large 4 (Le Chonk) is now open weight". Mistral AI, October 29, 2026. Accessed October 31, 2026.

Show all 39 references