# SkyRL

> Source: https://aiwiki.ai/wiki/skyrl
> Updated: 2026-10-30
> Fact-checked: 2026-10-30
> Categories: Developer Tools, Open Source AI, Reinforcement Learning, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "SkyRL." aiwiki.ai, 30 Oct 2026. https://aiwiki.ai/wiki/skyrl
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**SkyRL** is an open-source [reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning) (RL) library for post-training [large language models](https://aiwiki.ai/wiki/large_language_model) and multi-turn agents. It is developed by NovaSky, a student-and-advisor group at the Sky Computing Lab of the [University of California, Berkeley](https://aiwiki.ai/wiki/uc_berkeley), in collaboration with [Anyscale](https://aiwiki.ai/wiki/anyscale).[1][4] The code is published at github.com/NovaSky-AI/SkyRL under the [Apache License 2.0](https://aiwiki.ai/wiki/apache_license); the repository was created on April 22, 2025, and as of October 30, 2026 the GitHub API reported 2,476 stars, 472 forks and 115 contributors.[1][2] SkyRL began in May 2025 as a pipeline for training multi-turn, tool-using agents on long-horizon tasks such as software-engineering benchmarks, was rebuilt in mid-2025 as a modular framework, and has since become a single `skyrl` package with [FSDP](https://aiwiki.ai/wiki/fsdp), [Megatron](https://aiwiki.ai/wiki/megatron_lm) and [JAX](https://aiwiki.ai/wiki/jax) training backends, [vLLM](https://aiwiki.ai/wiki/vllm) for generation, and an implementation of the [Tinker](https://aiwiki.ai/wiki/tinker) training API.[1][3][10]

In October 2026 SkyRL gained "day-0" RL recipes for newly released open-weight models: one for [Beam](https://aiwiki.ai/wiki/beam_reflection) from [Reflection AI](https://aiwiki.ai/wiki/reflection_ai), published by the SkyRL team on October 16, and one for [Mistral Large 4](https://aiwiki.ai/wiki/mistral_large_4), which Mistral and the vLLM team announced on October 29, the day [Mistral AI](https://aiwiki.ai/wiki/mistral_ai) released that model's weights.[18][19][20][39] The Mistral Large 4 work was announced on the vLLM blog and on X, but on October 30, 2026 its code was still in five open, unmerged pull requests.[19][20][23]

## Background

RL post-training of a language model alternates between two jobs. An inference engine generates responses (rollouts) and records the probability of each sampled token; a training engine then scores those responses with a reward and updates the weights, which are copied back to the inference engine for the next round. The SkyRL-v0 announcement argued that most open RL frameworks at the time were tuned for short, stateless interactions such as search-augmented reasoning or single code executions, while tasks like [SWE-bench](https://aiwiki.ai/wiki/swe_bench) need long-horizon planning in stateful environments, which makes both the infrastructure and the training harder.[5]

A second problem became central to SkyRL's later work: the trainer and the inference engine compute slightly different probabilities for the same token because they use different kernels, batch shapes, parallel layouts and numeric formats. SkyRL's documentation lists kernel mismatch, inconsistent [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) routing and different parallelism settings as sources of this drift, and provides corrections for it.[13]

## Developers

NovaSky describes itself as "a collaborative initiative led by students and advisors at UC Berkeley's Sky Computing Lab". Its About page lists Dacheng Li, Shiyi Cao, Shu Liu and Tyler Griggs as student leads and Shishir G. Patil, Matei Zaharia, Joseph E. Gonzalez and [Ion Stoica](https://aiwiki.ai/wiki/ion_stoica) as advisors.[4] The SkyRL README says the work is done at the Berkeley Sky Computing Lab in collaboration with Anyscale, with compute support from Anyscale, [Databricks](https://aiwiki.ai/wiki/databricks), [NVIDIA](https://aiwiki.ai/wiki/nvidia), [Lambda](https://aiwiki.ai/wiki/lambda_labs), [AMD](https://aiwiki.ai/wiki/amd), AWS, Modal and Daytona.[1] The SkyRL-v0 author list combined UC Berkeley, Anyscale and All Hands AI researchers, and an October 2026 SkyRL blog post refers to "the SkyRL team from Anyscale".[5][30]

Before SkyRL, NovaSky released Sky-T1-32B-Preview (January 10, 2025), a reasoning model the group said it trained for under $450. Sky-T1 came from a separate repository, SkyThought, and was trained by [supervised fine-tuning](https://aiwiki.ai/wiki/fine_tuning) of [Qwen2.5](https://aiwiki.ai/wiki/qwen2_5)-32B-Instruct on 17,000 traces using Llama-Factory, not with SkyRL, which did not yet exist.[28] A follow-up 7B model, Sky-T1-7B, added RL stages using the PRIME and [RLOO](https://aiwiki.ai/wiki/rloo) algorithms.[37]

## Components

The README lists the following parts of the repository as of October 30, 2026.[1]

| Component | Description |
|---|---|
| `skyrl` | The unified library "for RL on your own hardware, with support for the Tinker API". It merged the earlier `skyrl-train` and `skyrl-tx` packages; its first release was skyrl v0.1.0 on March 11, 2026.[1][3] |
| `skyrl-train` | The original modular training framework for RL, released with SkyRL-v0.1 in June 2025, with its own tagged releases up to skyrl_train v0.4.0 (February 2026).[1][3] |
| `skyrl-tx` | A cross-platform backend for the Tinker API with "a unified engine for training and inference", first released October 6, 2025.[1] |
| `skyrl-agent` | An agent layer for training long-horizon agents; the docs call it SkyAgent, "a flexible frontend for building your own agents", and say it is "still under active development".[1][35] |
| `skyrl-gym` | A library of tool-use environments (math, coding, search and SQL) implemented with the Gymnasium API.[1] |

The repository also contains `skycap`, a separate package for capturing agent trajectories: an agent harness points its unchanged OpenAI-compatible client at a per-trajectory URL, and skycap records each model call and returns training samples.[38]

The repository's LICENSE file is Apache 2.0, while the `skyrl-gym` and `skyrl-train` packages declare the MIT license in their PyPI metadata.[1][33]

## Architecture

SkyRL's system overview splits training into a **Trainer**, which runs the optimization step for the configured algorithm, and a **Generator**, which produces complete trajectories and their rewards. The Generator in turn contains an **InferenceEngine** (model completions) and an **Environment** (task logic, observations and rewards), and a **Controller** handles placement, initialization and control flow. Training workers run as [Ray](https://aiwiki.ai/wiki/ray) actors.[8]

Version 0.2.0 of the unified package (April 2026) moved inference onto HTTP, and v0.3.0 removed the older code path. A `RemoteInferenceClient` sends routed generation requests through a `VLLMRouter` (a wrapper around vLLM's router) to standard vLLM API servers, and sends control requests (pause, resume, sleep, wake and weight sync) to every server directly.[3][9] Requests from one trajectory carry a session ID so that every turn lands on the same replica and reuses its prefix cache. A `sticky_least_loaded` policy, added in v0.3.0, sends new trajectories to the least-busy replica; SkyRL reported up to 1.4 times higher throughput on a multi-turn agentic task with GLM 4.7.[3][9] SkyRL uses vLLM's native weight-syncing and pause/resume APIs, which SkyRL and vLLM developers described together in May 2026.[31]

Version 0.4.0 (October 2026) added several weight-transfer paths: delta weight sync, which sends compressed XOR differences against the previous BF16 weights over disk, S3 or GCS-style storage for disaggregated clusters; sharded transfer over NIXL via Ray Direct Transport, which SkyRL says moved the BF16 weights of [Kimi K2](https://aiwiki.ai/wiki/kimi_k2) in 7.53 seconds across 48 nodes of eight [H100](https://aiwiki.ai/wiki/nvidia_h100) GPUs; and in-memory [LoRA](https://aiwiki.ai/wiki/lora) adapter sync over NCCL or CUDA IPC instead of through disk.[17] The framework supports colocated training, where trainer and inference share GPUs, as well as one-step off-policy and fully asynchronous training; fully asynchronous training arrived in skyrl_train v0.3.0 in December 2025.[1][3]

## Backends and hardware

| Layer | Supported options (October 2026) |
|---|---|
| Training | FSDP (PyTorch's `fully_shard`, i.e. FSDP2) is the default. Megatron provides 5D parallelism (data, [tensor](https://aiwiki.ai/wiki/tensor_parallelism), context, [pipeline](https://aiwiki.ai/wiki/pipeline_parallelism) and [expert](https://aiwiki.ai/wiki/expert_parallelism) parallel) through NVIDIA's Megatron-Bridge. A JAX backend runs through the Tinker API server.[10][11] |
| Inference | vLLM. [SGLang](https://aiwiki.ai/wiki/sglang) was supported by skyrl-train but dropped in skyrl v0.1.0 (March 2026), when the project unified on vLLM.[3] |
| Fine-tuning modes | Full-parameter training and LoRA; supervised fine-tuning through a native SFT trainer, which v0.4.0 extended to datasets of 10 billion to more than 1 trillion tokens, loaded from pretokenized files on disk.[12][17] |
| Precision | FP8 quantized rollouts with off-policy correction; blockwise FP8 and MXFP8 training with on-policy weight sync (v0.4.0).[17][34] |
| Hardware | NVIDIA GPUs; AMD GPUs via a ROCm Docker image built on vLLM's ROCm image for the Tinker API with the FSDP backend.[36] |

The FSDP backend accepts any model supported by Hugging Face Transformers and vLLM; the docs describe it as suited to dense models up to 32B parameters. The Megatron backend is meant for large MoE models above 100B parameters.[10] On October 30, 2026 the docs listed full Megatron support for Qwen3.5 (dense and MoE up to 397B-A17B), [Nemotron 3](https://aiwiki.ai/wiki/nemotron_3), GLM-4.7-Flash and GLM-4.7, [GLM-5.3-Flash](https://aiwiki.ai/wiki/glm_5_3_flash), Qwen3 (up to 235B-A22B) and Moonlight-16B-A3B, and JAX support for dense Qwen3.5, [DeepSeek-V3](https://aiwiki.ai/wiki/deepseek_v3), [Llama 3](https://aiwiki.ai/wiki/llama_3) and Qwen3.[10] The v0.4.0 notes added [GLM-5.3](https://aiwiki.ai/wiki/glm_5_3), [Qwen3.8](https://aiwiki.ai/wiki/qwen3_8) and [Nemotron 3.5 Lightning](https://aiwiki.ai/wiki/nemotron_3_5_lightning), plus experimental LoRA training of [Kimi K2.6](https://aiwiki.ai/wiki/kimi_k2_6) and K2.7, which SkyRL says fits a 1-trillion-parameter model on two B300 nodes by serving in INT4 and fake-quantizing the trainer.[17] Commits merged on October 30, 2026 added Megatron support for the text model of [DeepSeek V4.1-Flash](https://aiwiki.ai/wiki/deepseek_v4_1_flash) and for a model the commit calls Qwen3.8-Flash-Next.[36]

## Algorithms

SkyRL implements [PPO](https://aiwiki.ai/wiki/ppo)-style training with interchangeable advantage estimators and policy losses, and a registry for adding custom ones without changing the core code.[14] The source code on October 30, 2026 defined these built-in options:[15]

| Type | Built-in options |
|---|---|
| Advantage estimators | GAE (for PPO with a critic), [GRPO](https://aiwiki.ai/wiki/grpo), RLOO, REINFORCE++ and MaxRL |
| Policy losses | Clipped PPO ("regular"), dual-clip PPO, [GSPO](https://aiwiki.ai/wiki/gspo), CISPO, SAPO, Clip-Cov, KL-Cov, rollout importance sampling, DPPO, REINFORCE, cross-entropy (for SFT) and importance sampling (used by the Tinker API) |

SkyRL supports the [DAPO](https://aiwiki.ai/wiki/dapo) recipe (asymmetric "clip-higher" bounds, dynamic sampling that discards prompts whose samples all get the same reward, token-level loss averaging and overlong-response penalties) mostly through configuration; the soft overlong penalty is applied by a custom trainer class in SkyRL's DAPO example.[16] For trainer-inference mismatch, SkyRL offers truncated [importance sampling](https://aiwiki.ai/wiki/importance_sampling) (TIS) at token or sequence level, sequence masking by the geometric mean of importance ratios, token masking, rollout routing replay (R3) for MoE models, top-p and top-k sampler replay (v0.4.0) and score centering.[13][17] SkyRL's own comparison on a 1.5B distilled model found geometric sequence masking gave better evaluation results than TIS in one-step off-policy training.[13]

### Tinker API

SkyRL implements the Tinker API that [Thinking Machines Lab](https://aiwiki.ai/wiki/thinking_machines_lab) introduced, a small set of calls (`forward_backward`, `forward`, `optim_step`, `sample`, `save_checkpoint` and `load_checkpoint`). SkyRL says any script written against it runs on local hardware with "zero code changes", on FSDP or Megatron for training and vLLM for sampling.[12] The work started as SkyRL tx in October 2025, and skyrl_train v0.4.0 (February 13, 2026) brought full Tinker API support to the main training stack.[1][3] The docs list multi-tenant LoRA training and sampling as supported on Megatron with vLLM, but not yet on FSDP.[12]

## Projects and models

| Date | Project | Reported result |
|---|---|---|
| May 6, 2025 | SkyRL-v0 | Agents trained on around 300 examples with a pipeline built on veRL and [OpenHands](https://aiwiki.ai/wiki/openhands). The team reported [SWE-bench Verified](https://aiwiki.ai/wiki/swe_bench_verified) gains from 11.0% to 14.6% (SkyRL-Agent-7B-v0, from OpenHands-7B-Agent), 3.6% to 9.4% (SkyRL-Agent-8B-v0, from Qwen3-8B without thinking) and 18.0% to 21.6% (SkyRL-Agent-14B-v0, from Qwen3-14B with thinking).[5] |
| May 20, 2025 | SkyRL-SQL | A multi-turn Text-to-SQL pipeline in which the model builds and refines SQL queries using feedback from the database. The team said SkyRL-SQL-7B, trained on 653 examples, raised execution accuracy by up to 9.2% on five Spider benchmarks and outperformed GPT-4o, o4-mini and the SFT model OmniSQL-7B.[6] |
| November 20, 2025 | SkyRL-Agent and SA-SWE-32B | The SkyRL-Agent paper (arXiv:2511.16108) reported that SA-SWE-32B, trained from Qwen3-32B purely with RL, rose from 24.4% to 39.4% pass@1 on SWE-bench Verified, with an asynchronous dispatcher 1.55 times faster than naive asynchronous batching.[7] |
| September 1, 2026 | Mercor APEX-Agents recipe | [Mercor](https://aiwiki.ai/wiki/mercor) used SkyRL to post-train Qwen3.5-397B-A17B on 1,928 expert-written knowledge-work tasks and reported APEX-Agents pass@1 rising from 16.11% to 27.29%; APEX-Agents is Mercor's own benchmark.[25] |
| October 22, 2026 | GLM-5.3-Flash at 1M tokens | With [Lila Sciences](https://aiwiki.ai/wiki/lila_sciences), SkyRL reported training GLM-5.3-Flash on sequences of 1 million tokens, 64 times its previous baseline, using sparse-attention fixes, activation recompute and offloading, and context parallelism. The memory figures came from dummy runs on random tokens.[30] |

The checkpoints released with SkyRL-v0, SkyRL-SQL and SkyRL-Agent are hosted on the NovaSky-AI Hugging Face organization, while Mercor published its recipe under its own GitHub and Hugging Face accounts.[25][29] Outside groups have also built on the library: the CodeScout paper (March 2026) used SkyRL as its training backend for code-search agents, and the Endless Terminals paper (January 2026) trained terminal agents with PPO and states that "our implementation uses SkyRL".[26][27] The README lists further projects that use SkyRL, including the biomedical reasoning agent Biomni-R0 and [OpenThoughts](https://aiwiki.ai/wiki/openthoughts)-Agent.[1] The project also supports environments from Harbor (from the team behind [Terminal-Bench](https://aiwiki.ai/wiki/terminal_bench)) and [Prime Intellect](https://aiwiki.ai/wiki/prime_intellect)'s Environments Hub.[1][3][25]

In August 2026 Alexander Jiang and the SkyRL team described IsoExec, an opt-in execution contract with batch-invariant kernels that made vLLM and Megatron compute nearly identical log-probabilities in one 50-step test, at a reported 25% step-time cost; the [vLLM article](https://aiwiki.ai/wiki/vllm) covers it in detail.[32]

## Release history

| Date | Release | Main changes |
|---|---|---|
| May 6, 2025 | SkyRL-v0 | RL pipeline for multi-turn, tool-using agents on long-horizon tasks such as SWE-bench, built on veRL and OpenHands[1][5] |
| May 20, 2025 | SkyRL-SQL | Multi-turn Text-to-SQL training and the SkyRL-SQL-7B model[1][6] |
| June 26, 2025 | SkyRL-v0.1 and SkyRL-Gym | Rebuilt as "a highly-modular, performant RL training framework"; Gymnasium-style environments[1] |
| August 20, 2025 | skyrl_train v0.1.0 | First tagged GitHub release[3] |
| October 6, 2025 | SkyRL tx | Open implementation of a Tinker API backend[1] |
| October 13, 2025 | skyrl_train v0.2.0 | Megatron backend, LoRA with FSDP, OpenAI-compatible inference endpoints, mini-swe-agent and Environments Hub integrations[3] |
| December 3, 2025 | skyrl_train v0.3.0 | Fully asynchronous training; move from mbridge to NVIDIA Megatron-Bridge[3] |
| February 13, 2026 | skyrl_train v0.4.0 | Full Tinker API support, including the Megatron backend; docs moved to docs.skyrl.ai[3] |
| March 11, 2026 | skyrl v0.1.0 | Unified `skyrl` package; Python dataclass configuration replaces Hydra/YAML; SGLang support removed[3] |
| April 23, 2026 | skyrl v0.2.0 | Vision-language model training, HTTP-based inference with prefill-decode disaggregation, R3 for MoE models[3] |
| July 17, 2026 | skyrl v0.3.0 | Legacy inference path removed, `sticky_least_loaded` routing, dynamic micro-batching, dynamic sampling in fully asynchronous RL[3] |
| October 1, 2026 | skyrl v0.4.0 | GLM-5.3, Kimi K2.6/K2.7, Qwen3.8 and Nemotron 3.5 Lightning support; top-p sampler replay; faster R3; FP8 and MXFP8 training; delta and sharded weight sync; large-scale SFT[3][17] |

## Day-zero support for Beam (October 2026)

On October 16, 2026, the day Reflection AI released Beam's weights, SkyRL published LoRA, BF16 full fine-tuning and MXFP8 RL recipes for the 501B-parameter MoE model, trained with Megatron and served with vLLM. SkyRL called it "the first Day-0 RL support effort in SkyRL". On the DAPO recipe it reported AIME 2025 avg@8 accuracy rising from 61% to 79% in 50 training steps, and said the changes it made let Beam be trained with LoRA on as few as two B200 nodes.[18] The code lives on a separate `beam-release` branch.[1] The [Beam article](https://aiwiki.ai/wiki/beam_reflection) covers the runs and the engineering changes in detail.

## Day-zero support for Mistral Large 4 (October 2026)

Mistral AI released the weights of Mistral Large 4 (a 1-trillion-parameter MoE model with 128 routed experts, four active per token) on October 29, 2026.[19][39] The same day, the vLLM blog post "Day 0 support for Mistral Large 4 on vLLM", credited to "Mistral and the vLLM Team", included a section saying the teams had "enabled Day 0 RL for Mistral Large 4 in SkyRL".[19] A GitHub issue opened on the SkyRL repository on October 29, titled "[RFC] Day-0 RL support for Mistral Large 4 (NVFP4) on FSDP and Megatron", tracks the work and lists five stacked pull requests (#2576 to #2580).[20] The vLLM project's X account promoted the results on October 30.[23] As of October 30, 2026, the issue and all five pull requests were open, and the pull requests, all marked as drafts, had been submitted from a contributor's fork rather than merged into SkyRL's main branch; they were still open and unmerged on October 31.[20][21]

### Recipe

The run fine-tunes LoRA adapters on top of Mistral's NVFP4 checkpoint, with vLLM generating rollouts and either FSDP2 or Megatron training the adapters, on eight [NVIDIA GB200](https://aiwiki.ai/wiki/nvidia_gb200) nodes.[19] The example configuration in pull request #2576 describes that as eight nodes of four GPUs, or 32 GPUs in total, with training and inference colocated on the same GPUs.[21]

| Setting | Value in the published recipe |
|---|---|
| Base checkpoint | Mistral-Large-4-1T-A52B-NVFP4 (NVFP4 experts, FP8 attention and shared experts)[19][21] |
| Task and reward | [GRPO](https://aiwiki.ai/wiki/grpo) on DAPO-Math-17K with a rule-based math grader, a form of [RLVR](https://aiwiki.ai/wiki/rlvr); evaluation on [AIME](https://aiwiki.ai/wiki/aime) 2024 (deduplicated to 30 problems), mean accuracy over 8 samples[20][21] |
| Batch | 32 prompts per step, 8 samples per prompt, one optimizer update per batch; responses up to 4,096 tokens[21] |
| LoRA | Rank 32, alpha 64, on attention projections and on dense, shared and routed-expert MLP layers[21] |
| Optimization | Learning rate 1e-5; PPO clip range 0.2 below and 0.28 above; no KL penalty; token-level TIS capped at 2.0[21] |
| Rollouts | Four vLLM engines with 8-way tensor parallelism, FP8 latent KV cache, routed-expert capture for R3[21] |
| Megatron variant | Expert parallelism over 32 GPUs (TP1, PP1, EP32), colocated with four vLLM TP8 engines[19][20] |

The terms in the recipe mean the following:

- **LoRA (rank 32).** Low-rank adaptation freezes the original weights and trains small added matrices; the rank sets their size. Only these adapters are trained and copied to vLLM after each step.[21]
- **NVFP4 quantization-aware training.** [NVFP4](https://aiwiki.ai/wiki/nvfp4) is NVIDIA's 4-bit floating-point format for [Blackwell](https://aiwiki.ai/wiki/nvidia_blackwell) GPUs. In [quantization-aware training](https://aiwiki.ai/wiki/quantization_aware_training), the trainer reproduces the rounding the inference engine applies, so both compute the same numbers. According to the Megatron pull request, the expert weights stay in their NVFP4 form in memory, and the trainer applies NVFP4 and [FP8](https://aiwiki.ai/wiki/fp8) activation quantization that matches the rollout engine.[22]
- **FP8 KV-cache simulation.** Mistral Large 4 uses [multi-head latent attention](https://aiwiki.ai/wiki/multi-head_latent_attention), which stores a compressed "latent" version of each token's keys and values in the [KV cache](https://aiwiki.ai/wiki/kv_cache), and vLLM serves the model with that cache in FP8 by default. The trainer simulates the FP8 cache contents so its attention matches what vLLM computed during the rollout.[19][21]
- **Routing replay (R3).** In a mixture-of-experts layer a router picks which experts process each token. Small numeric differences can make the trainer pick different experts from those vLLM used, which changes the token probabilities. Rollout Routing Replay records vLLM's expert choices and forces the trainer to reuse them. It was introduced in October 2025 by researchers from Peking University and Xiaomi, who reported that it reduced training-inference divergence and prevented RL collapse in their MoE experiments.[13][24]
- **Token-level truncated importance sampling.** Because the tokens were sampled by vLLM, the trainer reweights each token's loss by the ratio of its own probability to vLLM's probability, capped at a maximum (2.0 here) so that a few extreme ratios cannot dominate the update. SkyRL's docs attribute the technique to a 2025 analysis by Yao and colleagues.[13][21]
- **Logprob gap.** The mean absolute difference between the log-probabilities that the trainer and vLLM assign to the same sampled tokens, used as a measure of how far apart the two systems are.[13]

### Reported results

All figures below are self-reported by the teams involved and have not been independently reproduced.

| Measure | FSDP2 run | Megatron run |
|---|---|---|
| AIME 2024 accuracy (mean of 8 samples) | 0.637 at step 0, 0.746 at step 75, then 0.72-0.73 at steps 95-100 in a 105-step run[20] | "follows the same accuracy curve", according to the vLLM post[19] |
| AIME 2024 pass@8 | Rose from 0.73 to 0.90[20] | Not reported |
| Mean absolute trainer-vLLM logprob gap | 0.019-0.026 for the whole run (issue); 0.02-0.026 (vLLM post)[19][20] | 0.021-0.027[20] |
| Tokens clipped by TIS | Under 0.1%[20] | Not reported |
| Step time | About 500 seconds[20] | About 445 seconds (issue); "about 30% faster than FSDP" (vLLM post)[19][20] |

The vLLM post describes the accuracy change as about 64% to about 75% "within 75 steps", and the X post says "in 75 training steps".[19][23] The tracking issue shows that the run continued past that point and that accuracy was slightly lower, 0.72 to 0.73, at steps 95-100.[20] The two sources also give different speed comparisons for the Megatron backend: the issue's step times correspond to a step roughly 11% shorter than FSDP's, measured "after the sync/offload/balancing work in this series", while the vLLM post says about 30%. Neither source explains the difference.[19][20]

## References

1. NovaSky-AI. ["SkyRL: A Modular Full-stack RL Library for LLMs"](https://github.com/NovaSky-AI/SkyRL). GitHub repository README and LICENSE. Accessed October 30, 2026.
2. GitHub REST API. [Repository record for NovaSky-AI/SkyRL](https://api.github.com/repos/NovaSky-AI/SkyRL) and [contributors list](https://api.github.com/repos/NovaSky-AI/SkyRL/contributors). Accessed October 30, 2026.
3. NovaSky-AI. [SkyRL releases](https://github.com/NovaSky-AI/SkyRL/releases): skyrl_train v0.1.0 to v0.4.0 and skyrl v0.1.0 to v0.4.0 release notes. GitHub. Accessed October 30, 2026.
4. NovaSky. ["About Us"](https://novasky-ai.github.io/posts/about-us/). Sky Computing Lab, UC Berkeley. Accessed October 30, 2026.
5. Cao, Shiyi, Sumanth Hegde, Dacheng Li, Tyler Griggs, Shu Liu, Eric Tang, et al. ["SkyRL-v0: Train Real-World Long-Horizon Agents via Reinforcement Learning"](https://novasky-ai.notion.site/skyrl-v0). NovaSky blog, May 2025. Accessed October 30, 2026.
6. Liu, Shu, Sumanth Hegde, Shiyi Cao, Alan Zhu, et al. ["SkyRL-SQL: Matching GPT-4o and o4-mini on Text2SQL with Multi-Turn RL"](https://novasky-ai.notion.site/skyrl-sql). NovaSky blog, May 20, 2025. Accessed October 30, 2026.
7. Cao, Shiyi, Dacheng Li, Fangzhou Zhao, et al. ["SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent"](https://arxiv.org/abs/2511.16108). arXiv:2511.16108, November 20, 2025.
8. SkyRL Documentation. ["SkyRL System Overview"](https://docs.skyrl.ai/docs/getting-started/overview). Accessed October 30, 2026.
9. SkyRL Documentation. ["Inference Architecture"](https://docs.skyrl.ai/docs/getting-started/inference_architecture). Accessed October 30, 2026.
10. SkyRL Documentation. ["Supported Models"](https://docs.skyrl.ai/docs/getting-started/supported_models). Accessed October 30, 2026.
11. SkyRL Documentation. ["Switching Training Backends"](https://docs.skyrl.ai/docs/examples/training_backends). Accessed October 30, 2026.
12. SkyRL Documentation. ["Tinker API"](https://docs.skyrl.ai/docs/tinker/overview). Accessed October 30, 2026.
13. SkyRL Documentation. ["Off-Policy Correction"](https://docs.skyrl.ai/docs/algorithms/off_policy_correction). Accessed October 30, 2026.
14. SkyRL Documentation. ["Implementing Custom Algorithms"](https://docs.skyrl.ai/docs/algorithms/custom_algorithms). Accessed October 30, 2026.
15. NovaSky-AI. [`ppo_utils.py`](https://github.com/NovaSky-AI/SkyRL/blob/main/skyrl/backends/skyrl_train/utils/ppo_utils.py) (AdvantageEstimator and PolicyLossType definitions). SkyRL main branch. Accessed October 30, 2026.
16. SkyRL Documentation. ["DAPO"](https://docs.skyrl.ai/docs/algorithms/dapo). Accessed October 30, 2026.
17. SkyRL Blog. ["SkyRL v0.4.0 Release"](https://blog.skyrl.ai/blog/skyrl-v0-4-0). October 6, 2026. Accessed October 30, 2026.
18. Hegde, Sumanth, and the SkyRL team. ["Day 0 Support for Beam in SkyRL"](https://blog.skyrl.ai/blog/day-0-beam). SkyRL Blog, October 16, 2026. Accessed October 30, 2026.
19. Mistral and the vLLM Team. ["Day 0 support for Mistral Large 4 on vLLM"](https://vllm.ai/blog/2026-10-29-mistral-large-4), section "Day 0 Support for RL with SkyRL". vLLM Blog, October 29, 2026. Accessed October 30, 2026.
20. NovaSky-AI/SkyRL issue #2575. ["[RFC] Day-0 RL support for Mistral Large 4 (NVFP4) on FSDP and Megatron"](https://github.com/NovaSky-AI/SkyRL/issues/2575). GitHub, opened October 29, 2026. Accessed October 30, 2026.
21. NovaSky-AI/SkyRL pull request #2576. ["feat(fsdp): ML4 (Mistral Large 4 NVFP4) LoRA RL on FSDP2"](https://github.com/NovaSky-AI/SkyRL/pull/2576), including `examples/ml4/ml4_dapo_lora_fsdp.yaml`. GitHub, opened October 29, 2026. Accessed October 30, 2026.
22. NovaSky-AI/SkyRL pull request #2577. ["feat(megatron): ML4 bridge, layer spec and modules with NVFP4/FP8 QAT"](https://github.com/NovaSky-AI/SkyRL/pull/2577). GitHub, opened October 29, 2026. Accessed October 30, 2026.
23. vLLM (@vllm_project). [Post on day-0 RL for Mistral Large 4 with SkyRL and vLLM](https://x.com/vllm_project/status/2115985388649242699). X, October 30, 2026.
24. Ma, Wenhan, Hailin Zhang, Liang Zhao, Yifan Song, Yudong Wang, Zhifang Sui, and Fuli Luo. ["Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers"](https://arxiv.org/abs/2510.11370). arXiv:2510.11370, October 13, 2025 (v2 October 21, 2025).
25. Mercor and SkyRL. ["Training frontier knowledge work agents: A 397B RL training guide with SkyRL"](https://www.mercor.com/blog/training-frontier-knowledge-work-agents-a-397b-rl-training-guide-with-skyrl/). Mercor Blog, September 1, 2026. Accessed October 30, 2026.
26. Sutawika, Lintang, et al. ["CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents"](https://arxiv.org/abs/2603.17829). arXiv:2603.17829, March 18, 2026.
27. Gandhi, Kanishk, Shivam Garg, Noah D. Goodman, and Dimitris Papailiopoulos. ["Endless Terminals: Scaling RL Environments for Terminal Agents"](https://arxiv.org/abs/2601.16443). arXiv:2601.16443, January 23, 2026.
28. NovaSky Team. ["Sky-T1: Train your own O1 preview model within $450"](https://novasky-ai.github.io/posts/sky-t1/). NovaSky blog, January 10, 2025. Accessed October 30, 2026.
29. NovaSky-AI. [Hugging Face organization](https://huggingface.co/NovaSky-AI) (SkyRL-Agent-7b-v0, SkyRL-Agent-8b-v0, SkyRL-Agent-14B-v0, SkyRL-SQL-7B, SA-SWE-32B and datasets). Accessed October 30, 2026.
30. Basnet, Avi, and the SkyRL Team; Keith Truongcao and the Lila Sciences AI Research Team. ["Scaling Training to 1M Token Context on GLM-5.3-Flash"](https://blog.skyrl.ai/blog/1m-context-glm-5-3-flash). SkyRL Blog, October 22, 2026. Accessed October 30, 2026.
31. Hao, Aaron, Sumanth Hegde, Kyle Sayers, Kourosh Hakhamaneshi, and the vLLM team. ["Native RL APIs in vLLM"](https://vllm.ai/blog/2026-05-28-native-rl-apis). vLLM Blog, May 28, 2026. Accessed October 30, 2026.
32. Jiang, Alexander, and the SkyRL Team. ["IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL"](https://vllm.ai/blog/2026-08-21-isoexec). vLLM Blog, August 21, 2026. Accessed October 30, 2026.
33. Python Package Index. [skyrl-gym](https://pypi.org/project/skyrl-gym/) and [skyrl-train](https://pypi.org/project/skyrl-train/) package metadata; see also [`skyrl-gym/pyproject.toml`](https://github.com/NovaSky-AI/SkyRL/blob/main/skyrl-gym/pyproject.toml). Accessed October 30, 2026.
34. SkyRL Documentation. ["Training with Quantized Rollouts"](https://docs.skyrl.ai/docs/examples/quantized_rollouts). Accessed October 30, 2026.
35. SkyRL Documentation. ["SkyAgent Overview"](https://docs.skyrl.ai/docs/skyagent/agent-overview). Accessed October 30, 2026.
36. NovaSky-AI. [SkyRL commit history](https://github.com/NovaSky-AI/SkyRL/commits/main) (including pull requests #2567 and #2513) and SkyRL Documentation, ["AMD ROCm Tinker"](https://docs.skyrl.ai/docs/examples/amd). Accessed October 30, 2026.
37. NovaSky Team. ["Unlocking the Potential of Reinforcement Learning in Improving Reasoning Models"](https://novasky-ai.github.io/posts/sky-t1-7B/). NovaSky blog, February 13, 2025. Accessed October 30, 2026.
38. NovaSky-AI. [`skycap` README](https://github.com/NovaSky-AI/SkyRL/tree/main/skycap). SkyRL main branch. Accessed October 30, 2026.
39. Mistral AI. ["Mistral Large 4 (Le Chonk) is now open weight"](https://mistral.ai/news/mistral-large-4-oss/). Mistral AI, October 29, 2026. Accessed October 31, 2026.