# TorchTitan

> Source: https://aiwiki.ai/wiki/torchtitan
> Updated: 2026-10-16
> Fact-checked: 2026-10-16
> Categories: Developer Tools, Meta AI, Open Source AI, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "TorchTitan." aiwiki.ai, 16 Oct 2026. https://aiwiki.ai/wiki/torchtitan
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**TorchTitan** (styled `torchtitan` in its repository) is an open-source, [PyTorch](https://aiwiki.ai/wiki/pytorch)-native platform for training large generative AI models, developed in the `pytorch/torchtitan` GitHub repository under a BSD 3-Clause license.[1] It began as a minimal reference codebase for showcasing PyTorch's distributed training features on [Llama](https://aiwiki.ai/wiki/llama)-style language models and has since grown into a single stack that covers pretraining, supervised fine-tuning and reinforcement learning.[1][4] The project's design centers on composing several forms of parallelism (fully sharded data parallelism, tensor, pipeline, context and expert parallelism) on top of PyTorch's DTensor and DeviceMesh abstractions without rewriting the model code.[1][2] An accompanying paper, "TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining," written by engineers at Meta with collaborators from Harvard University, was accepted as a poster at [ICLR](https://aiwiki.ai/wiki/iclr) 2025.[2][3]

## History

The first commit to the repository, titled "first commit, starter files," was made by Wanchao Liang on 14 December 2023.[18] The project was made public on 16 April 2024, the same day version 0.0.2 was uploaded to PyPI. The README published with that release described TorchTitan as "currently in a pre-release state and under extensive development" and said it showcased pre-training of Llama 2 models from scratch, with FSDP2, tensor parallelism, selective activation checkpointing and distributed checkpointing as its key features.[4] Asynchronous checkpointing, FP8 support, context parallelism, pipeline parallelism and `torch.compile` support were listed as coming soon.[4]

That early README framed the project narrowly. It called TorchTitan "a proof-of-concept for Large-scale LLM training using native PyTorch," described it as "complementary to and not a replacement for" codebases such as [Megatron](https://aiwiki.ai/wiki/megatron_lm), Megablocks, LLM Foundry and [DeepSpeed](https://aiwiki.ai/wiki/deepspeed), and stated that it was "unlikely to ever grow a large community around it."[4] The current README instead calls it "a clean-room, PyTorch-native implementation of large-scale training" that takes a model "from pretraining through supervised fine-tuning to reinforcement learning."[1] As of 17 October 2026 the repository had about 5,800 stars and 1,000 forks on GitHub.[1]

The paper was first posted to arXiv on 9 October 2024 and revised in November 2024 and June 2025.[2] Torchforge, a PyTorch-native agentic reinforcement-learning library in the meta-pytorch GitHub organization that required TorchTitan, Monarch and vLLM, now carries a notice that its development has paused and that "LLM training at PyTorch is being consolidated in torchtitan."[15]

| Date | Event |
|---|---|
| 14 December 2023 | First commit to `pytorch/torchtitan`[18] |
| 16 April 2024 | Repository made public; version 0.0.2 on PyPI[4] |
| 9 October 2024 | Paper posted to arXiv (2410.06511)[2] |
| November 2024 | IBM and Meta publish float8 training results using TorchTitan[9] |
| 22 January 2025 | Paper accepted at ICLR 2025; presented as a poster at the conference, held 24-28 April 2025[3][19] |
| 18 June 2025 | v0.1.0, the first tagged pre-release[5] |
| 18 October 2025 | v0.2.0[5] |
| 10 November 2025 | vLLM and TorchTitan teams describe bitwise-consistent RL training[12] |
| 26 December 2025 | v0.2.1[5] |
| 20 February 2026 | v0.2.2[5] |
| August 2026 | TitanRL reinforcement-learning stack announced in the README[1] |
| 3 September 2026 | v0.3.0[6] |

Each tagged release pins the PyTorch and torchao nightly builds it was validated against. v0.1.0 listed `torch-2.8.0.dev20250617`, while v0.2.2 listed `torch-2.12.0.dev20260220`.[5] The README recommends installing from source with a recent PyTorch nightly, and also offers stable releases through pip and conda-forge.[1]

## Design

### Composable parallelism

The paper identifies the absence of "unified tensor and device abstractions applied consistently across the stack" as the reason existing distributed training systems were hard to compose and extend.[2] TorchTitan's answer was to build on PyTorch's Distributed Tensor (DTensor) and DeviceMesh, which describe how each tensor is sharded across a multi-dimensional grid of devices. The authors say they extended DTensor to support n-D sharding, work with `torch.compile`, and save n-D models through state-dict support.[2]

The codebase keeps three components as separate as possible: a model definition that is "parallelism-agnostic and designed for readability," parallelism helpers that apply parallelisms and optimizations to a particular model, and a generalized training loop.[2] In the paper-era code these were configured through TOML files with command-line overrides.[2] Release 0.3.0 replaced that with typed Python configuration, so a training run is a Python "recipe" function that returns a complete config and can be inherited and modified.[6][1]

Large models are first instantiated on PyTorch's meta device, which stores only metadata, and then sharded into DTensors before parameters are initialized; the paper says this avoids exhausting CPU or GPU memory before training begins.[2]

In August 2026 the project made `spmd_types` its default backend for distributed model computation. A PyTorch DevLog post explaining the change said that DTensor's implicit sharding propagation had caused runtime overhead and "surprising sharding decisions" as model support grew, and that the new backend has model authors specify sharding contracts and collectives explicitly. It reported that `spmd_types` improved eager throughput by up to 46% and Inductor throughput by 5-9% over the previous `partial_dtensor` backend on repeated debug-model benchmarks, while FSDP2 still uses DTensor as its storage representation.[8]

### Parallelism techniques

| Technique | How TorchTitan uses it |
|---|---|
| [Fully Sharded Data Parallel](https://aiwiki.ai/wiki/fsdp) (FSDP2) | Default 1D parallelism. FSDP2 shards each parameter as a DTensor rather than using FSDP1's FlatParameter; the paper reports often 7% lower per-GPU memory and an average 1.5% speed gain over FSDP1. HSDP and DDP are also supported.[2][1] |
| [Tensor parallelism](https://aiwiki.ai/wiki/tensor_parallelism) | Implemented with PyTorch's RowwiseParallel and ColwiseParallel APIs, bundled with sequence parallelism and loss parallelism; does not require changes to model code.[2] |
| Asynchronous tensor parallelism | Splits TP matrix multiplications into chunks and overlaps communication with computation, using PyTorch's SymmetricMemory; aimed at H100-class or newer GPUs with NVSwitch.[2] |
| [Pipeline parallelism](https://aiwiki.ai/wiki/pipeline_parallelism) | Built on `torch.distributed.pipelining`, with schedules including 1F1B, Interleaved 1F1B, GPipe, ZeroBubble and Flexible-Interleaved-1F1B.[2][1] |
| [Context parallelism](https://aiwiki.ai/wiki/context_parallelism) | Splits the sequence dimension across GPUs for long-context training; current variants are all-gather KV and Ulysses.[2][1] |
| [Expert parallelism](https://aiwiki.ai/wiki/expert_parallelism) | For [mixture-of-experts](https://aiwiki.ai/wiki/mixture_of_experts) models, with DeepEP and HybridEP dispatch; v0.3.0 reorganized MoE around a unified token-dispatcher interface.[1][6] |

### Memory, precision and compilation

TorchTitan offers full, op-level selective and layer-level selective activation checkpointing, letting users trade recomputation for memory.[2] It applies [torch.compile](https://aiwiki.ai/wiki/torch_compile) regionally, to each Transformer block separately. The paper gives two reasons: each region compiles to a full graph that works with FSDP2 and tensor parallelism, and identical stacked blocks only need to be compiled once, which cuts compilation time.[2]

For low precision, the paper describes [FP8](https://aiwiki.ai/wiki/fp8) (Float8) training of linear layers through torchao, with dynamic, delayed and static per-tensor scaling, composable with FSDP2 and tensor parallelism including Float8 all-gather.[2] The current feature list names MXFP8 and [NVFP4](https://aiwiki.ai/wiki/nvfp4) training, FlashAttention 3 and 4, FlexAttention, variable-length attention, BF16 optimizer states and FSDP CPU offload.[1]

### Checkpointing and debugging

Checkpoints use PyTorch Distributed Checkpointing (DCP), which stores DTensor shards in a parallelism-independent format so a checkpoint can be reloaded under a different parallel layout. The paper says asynchronous DCP saves cut checkpointing overhead by 5-15x compared with synchronous distributed checkpointing for Llama 3.1 8B.[2] The current version can also load and save Hugging Face [safetensors](https://aiwiki.ai/wiki/safetensors) directly.[1] For debugging, TorchTitan integrates PyTorch's Flight Recorder, which logs every collective and point-to-point operation to diagnose hung or crashed [NCCL](https://aiwiki.ai/wiki/nccl) jobs, and the current feature list adds fake-backend dry runs, a deterministic mode, loss comparison across commits and silent-data-corruption replay.[2][1]

## Paper results

The authors evaluated TorchTitan on the [Llama 3.1](https://aiwiki.ai/wiki/llama_3_1) 8B, 70B and 405B models on up to 512 [NVIDIA H100](https://aiwiki.ai/wiki/nvidia_h100) GPUs, training on the C4 dataset.[2] They noted that their H100s were non-standard, with HBM2e memory and a lower power limit, so peak TFLOPS were somewhere between the SXM and NVL versions.[2] They did not report model FLOPS utilization for Float8 runs, because both BF16 and FP8 tensor cores were in use, but said 1D Llama 3.1 8B training without Float8 reached 33% to 42% MFU on 8 or 128 GPUs.[2]

| Model and scale | Parallelism | Change measured | Throughput gain (per GPU) |
|---|---|---|---|
| Llama 3.1 8B, 8 GPUs | FSDP | Adding torch.compile and Float8 | +50.35% (6,258 to 9,409 tokens/s)[2] |
| Llama 3.1 8B, 128 GPUs | FSDP | Adding torch.compile and Float8 | +65.08% (5,645 to 9,319 tokens/s)[2] |
| Llama 3.1 70B, 256 GPUs | FSDP + TP (degree 32 x 8), with compile and Float8 | Adding asynchronous TP | +12.59% (897 to 1,010 tokens/s)[2] |
| Llama 3.1 405B, 512 GPUs | FSDP + TP + PP (4 x 8 x 16) | Interleaved 1F1B instead of 1F1B | +30.00% (100 to 130 tokens/s)[2] |

The paper also showed context parallelism extending trainable sequence length. On Llama 3.1 8B with 8 GPUs, moving from no context parallelism to a CP degree of 8 raised the sequence length from 32,768 to 262,144 tokens.[2] With four parallel dimensions combined on 512 GPUs, the 405B model was trained at sequence lengths up to 262,144 tokens.[2]

The paper's related-work section compares TorchTitan with Megatron-LM, DeepSpeed and ByteDance's veScale. The authors argue that Megatron-LM requires model changes for TransformerEngine and lacks seamless FSDP integration with TP and PP, that DeepSpeed depends on Megatron-LM for TP and CP, and that veScale lacks FSDP, CP and Float8 support. They also say each library "has its own strengths."[2]

## Supported models

The project README lists the following model families as of October 2026. Every model has debug configurations exercised in CI; "verified recipes" have been run in training, convergence or performance workflows.[1]

| Model family | Sizes listed | Modality | RL support in TitanRL |
|---|---|---|---|
| [Llama 3](https://aiwiki.ai/wiki/llama_3) | 1B to 405B | Text | Yes |
| [Qwen3](https://aiwiki.ai/wiki/qwen_3) | 0.6B to 32B dense; 30B-A3B and 235B-A22B MoE | Text | Yes |
| Qwen3.5 / 3.6 / 3.8 | 0.8B to 397B-A17B; 2.4T-A95B (3.8) | Text, vision | Text only |
| [DeepSeek V3](https://aiwiki.ai/wiki/deepseek_v3) | 16B, 236B, 671B | Text | No |
| [DeepSeek V4](https://aiwiki.ai/wiki/deepseek_v4) | Flash, Pro | Text | Listed as coming soon |
| [gpt-oss](https://aiwiki.ai/wiki/gpt_oss) | 20B, 120B | Text | Yes |
| Kimi K2.7 | Moonlight-16B-A3B, Kimi-VL-A3B, K2.5 | Text, vision | No |
| [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) | K3 | Text, vision | Listed as coming soon |
| [Muse Glimmer](https://aiwiki.ai/wiki/muse_glimmer) | 30B | Text, vision | Yes |
| [Muse Spark Open](https://aiwiki.ai/wiki/muse_spark_open) | 816B-A42B | Text, vision, audio | Text only |
| [FLUX](https://aiwiki.ai/wiki/flux) | dev, schnell | Text-to-image | No |

The repository's `experiments` folder hosts work outside the core, including GraphTrainer, TorchFT fault tolerance and a Transformers modeling backend.[1] Release 0.3.0 described GraphTrainer as capturing the forward, loss and backward step as one graph that can be precompiled once and reused across ranks.[6]

## TitanRL and post-training

TitanRL, announced in the README in August 2026, is TorchTitan's [reinforcement learning](https://aiwiki.ai/wiki/reinforcement_learning) stack. Its main idea is to run (mostly) the same TorchTitan model definition inside both the trainer and [vLLM](https://aiwiki.ai/wiki/vllm) generation; the README says the attention modules differ but use the same underlying kernels, so new models and kernels usually need to be written once.[7] In a batch-invariant mode, supported configurations produce bitwise-identical log probabilities across different data-parallel layouts, which the project presents as a way to debug correctness during on-policy training. The README notes that this mode trades throughput for exact numerics.[7] Rollouts run asynchronously, orchestrated by Monarch, and model weights are pushed from trainer to generators through TorchStore. The stack ships GRPO and DAPO losses with DAPO Math, Search-R1 and Verifiers examples.[1][7]

TitanRL grew out of earlier work. In November 2025 the vLLM and TorchTitan teams published an RL fine-tune of Qwen3 1.7B in which TorchTitan training and vLLM inference matched bitwise. They reported that this exact-match run was 2.4x slower than the non-bitwise case and listed a unified model definition as a future direction.[12]

## Adoption and integrations

| User or integration | Use of TorchTitan |
|---|---|
| IBM Research and Meta | A November 2024 PyTorch blog post used TorchTitan as the training entry point for float8 experiments with FSDP2, reporting up to 50% throughput speedup over FSDP1 bf16 training on Llama-architecture models from 1.8B to 405B parameters.[9] |
| [Nous Research](https://aiwiki.ai/wiki/nous_research) | The [Hermes 4](https://aiwiki.ai/wiki/hermes_4) technical report states that the models were trained with "a modified version of TorchTitan," starting from Llama 3.1 405B and 70B and Qwen3 14B checkpoints.[11] |
| PyTorch with [Crusoe](https://aiwiki.ai/wiki/crusoe) | A March 2026 PyTorch blog post reported a 30.2% training speedup for [Llama 4](https://aiwiki.ai/wiki/llama_4) Scout using MXFP8 MoE training in torchao, with TorchTitan as the training framework on a 256-GPU GB200 cluster on Crusoe Cloud.[10] |
| verl | The verl RL library offers a TorchTitan training engine that delegates model building, parallelization, optimizer sharding and checkpointing to TorchTitan; its docs say pipeline parallelism is not yet supported in that engine.[13] |
| [AMD](https://aiwiki.ai/wiki/amd) | AMD released an optimized fork of TorchTitan for AMD GPUs in November 2025, according to the TorchTitan README.[1][16] |
| SkyPilot | Added TorchTitan support with a tutorial in October 2025.[1][17] |
| [Zephon](https://aiwiki.ai/wiki/zephon) | A [DatologyAI](https://aiwiki.ai/wiki/datology_ai) data loader whose TorchTitan integration replaces the default data loader so that a run can resume on a different number of GPUs with identical global batches.[14] |

## References

1. pytorch/torchtitan. "torchtitan" (README). GitHub. https://github.com/pytorch/torchtitan
2. Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, Stratos Idreos. "TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training." arXiv:2410.06511, 2024 (v3 June 2025). https://arxiv.org/abs/2410.06511
3. ICLR 2025. "TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining" (poster). https://iclr.cc/virtual/2025/poster/29620
4. PyPI. "torchtitan 0.0.2." April 16, 2024. https://pypi.org/project/torchtitan/0.0.2/
5. pytorch/torchtitan. "Releases." GitHub. https://github.com/pytorch/torchtitan/releases
6. pytorch/torchtitan. "v0.3.0" release notes. GitHub, September 3, 2026. https://github.com/pytorch/torchtitan/releases/tag/v0.3.0
7. pytorch/torchtitan. "TitanRL" (README). GitHub. https://github.com/pytorch/torchtitan/blob/main/torchtitan/rl/README.md
8. Pian Pawakapan. "SPMD types in TorchTitan." PyTorch DevLog, August 26, 2026. https://docs.pytorch.org/devlogs/distributed/2026-08-26-spmd-types-in-torchtitan/
9. IBM and Meta. "Supercharging Training using float8 and FSDP2." PyTorch Blog, November 25, 2024. https://pytorch.org/blog/training-using-float8-fsdp2/
10. Daniel Vega-Myhre et al. "MXFP8 Training for MoEs: 1.3x training speedup vs BF16 for Llama4 Scout on GB200 cluster using TorchAO and TorchTitan." PyTorch Blog, March 12, 2026. https://pytorch.org/blog/mxfp8-training-for-moes-1-3x-training-speedup-vs-bf16-for-llama4-scout-on-gb200-cluster-using-torchao-and-torchtitan/
11. Ryan Teknium et al. "Hermes 4 Technical Report." arXiv:2508.18255, 2025. https://arxiv.org/abs/2508.18255
12. vLLM and TorchTitan Teams. "No More Train-Inference Mismatch: Bitwise Consistent On-Policy Reinforcement Learning with vLLM and TorchTitan." vLLM Blog, November 10, 2025. https://vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference
13. verl documentation. "TorchTitan Backend." https://verl.readthedocs.io/en/latest/workers/torchtitan_workers.html
14. Josh Wills, Maximilian Böther, Ties Robroek and the Zephon team. "Bringing Elastically Deterministic Data Loading to TorchTitan with Zephon." PyTorch Blog, October 16, 2026. https://pytorch.org/blog/bringing-elastically-deterministic-data-loading-to-torchtitan-with-zephon/
15. meta-pytorch/forge. "torchforge" (README). GitHub. https://github.com/meta-pytorch/forge
16. AMD-AGI/torchtitan-amd. GitHub. https://github.com/AMD-AGI/torchtitan-amd
17. SkyPilot documentation. "TorchTitan" training example. https://docs.skypilot.co/en/latest/examples/training/torchtitan.html
18. pytorch/torchtitan. "first commit, starter files" (commit 2d1c6d5). GitHub, December 14, 2023. https://github.com/pytorch/torchtitan/commit/2d1c6d53bc57cf38a6e5cc0ea1e6a2f4fd37a4fd
19. ICLR 2025. "Dates". International Conference on Learning Representations. https://iclr.cc/Conferences/2025/Dates