torch.compile

torch.compile is the just-in-time graph capture and compilation feature introduced in PyTorch 2.0, a release first announced at the PyTorch Conference on December 2, 2022 and shipped as a stable version on March 15, 2023.^[1][2] Wrapping a model with a single call (torch.compile(model)) routes its Python execution through a stack of new components, TorchDynamo, AOTAutograd, PrimTorch, and TorchInductor, that lift unmodified eager-mode code into optimized Triton kernels on NVIDIA A100-class GPUs or C++/OpenMP code on CPUs.^[1][3] Meta and the PyTorch Foundation reported geometric-mean training speedups of 38% on TIMM, 52% on HuggingFace Transformers, and 76% on TorchBench at the December 2022 announcement, with a follow-on ASPLOS 2024 paper measuring 2.27x inference and 1.41x training speedup across 180+ models on an A100.^[1][3] The design preserves PyTorch's "eager by default" programming model while introducing optional ahead-of-time compilation, an approach Meta describes as the largest architectural shift in PyTorch since the 1.0 release.^[1][4]

Infobox

Scroll sideways for more →

Background

PyTorch was originally a "define-by-run" framework: each tensor operation executes immediately in the Python interpreter, which is the eager mode that made the library popular among researchers but historically slower than graph-based frameworks like TensorFlow 1 or XLA-backed systems.^[4] Several earlier attempts to add compilation to PyTorch (TorchScript, FX tracing, Lazy Tensor, and various nvFuser/NNC fusers) each captured only part of a model or required users to rewrite code, and none became the default path.^[3]

Meta began the work that became torch.compile as TorchDynamo in 2020, prototyped by Jason Ansel; the ASPLOS 2024 paper describes the system as the outcome of a roughly five-year effort to find a graph-capture mechanism robust enough to work on arbitrary PyTorch programs.^[3][5] The official PyTorch 2.0 retrospective frames TorchDynamo's bytecode-rewriting approach as "the result of 5 years of R&D into safe graph capture."^[1]

The public roll-out followed a deliberate sequence:

• December 2, 2022. PyTorch 2.0 unveiled at the PyTorch Conference in New Orleans, with torch.compile available in nightly builds and a stable release promised for early March 2023.^[1][6]
• March 15, 2023. PyTorch 2.0 released as a stable version, accompanied by the official blog post "PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever."^[2]
• October 2023. PyTorch 2.1 ships at the PyTorch Conference 2023, adding automatic dynamic shape support to torch.compile, NumPy API support inside compiled regions, AVX-512 codegen on CPU, and improved Inductor schedulers.^[18]
• January 30, 2024. PyTorch 2.2 ships with FlashAttention-2 integration in scaled_dot_product_attention, the AOTInductor ahead-of-time deployment path, optimizer compilation improvements, horizontal fusion for torch.cat, and the unified TORCH_LOGS logging system.^[19]
• April 27 to May 1, 2024. The 50+-author paper "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation" presented at the 29th ACM ASPLOS conference in San Diego, with the proceedings published in early 2024.^[3][5]

PyTorch 2.0 is fully backward-compatible with PyTorch 1.x: torch.compile is an opt-in wrapper, not a new programming model, and code that does not call it continues to execute eagerly with no changes.^[1][2] The PyTorch Foundation, the Linux Foundation-hosted body that took stewardship of the project in late 2022, presents torch.compile as the flagship feature of the 2.x line and the central piece around which subsequent releases organize their performance work.^[4][19]

How It Works

torch.compile(fn) returns a callable that, on first invocation, traces and compiles the underlying Python function or nn.Module. The stack underneath is layered, with each component responsible for a different stage of lowering a Python program down to device code.^[1][3]

TorchDynamo (graph capture)

TorchDynamo is a Python-level just-in-time compiler that hooks into the CPython frame evaluation API specified by PEP 523.^[7][8] When a wrapped function is called, Dynamo intercepts the frame, walks the function's bytecode, and rewrites it on the fly: PyTorch operations are extracted into an FX graph (torch.fx.GraphModule), while constructs that cannot be safely captured (data-dependent control flow, arbitrary Python side effects, calls to C extensions) cause a "graph break" and fall back to the regular CPython interpreter.^[3][9]

Because Dynamo operates on bytecode rather than the Python source, it can capture code paths that traditional symbolic tracing cannot reach, including if/else branches, list comprehensions, and many third-party library calls. The ASPLOS 2024 paper reports that Dynamo captures graphs more robustly than prior PyTorch approaches while adding minimal overhead.^[3] On a 7,000-plus repository GitHub corpus, Meta reports a 99% graph-capture rate, with the remaining 1% falling back to eager execution.^[1]

PEP 523, accepted into Python 3.6, added a hook in the CPython interpreter that lets external C code substitute its own frame-evaluation function in place of _PyEval_EvalFrameDefault. Originally motivated as a generic JIT entry point, it became the foundation TorchDynamo uses to intercept Python execution without modifying the user's source code or requiring AST-level analysis.^[7] When TorchDynamo is active it installs a custom frame evaluator that, for each Python function call, decides whether to rewrite the bytecode (extracting an FX graph and substituting calls to the compiled artifact) or to defer to the default evaluator. Cached compilations are keyed on the bytecode object plus the guard set, so successive calls with the same shapes and types reuse the existing compiled artifact at near-zero overhead.^[7][9]

Guard-based recompilation

Each compiled graph is paired with a set of "guards", runtime checks that confirm the assumptions Dynamo made during tracing (tensor shapes, dtypes, attribute values, Python types) still hold for a new call.^[9][10] If all guards pass the cached compiled artifact is reused; if a guard fails, Dynamo retraces and produces a new specialization, optionally widening the assumption (for example, converting a fixed shape into a symbolic one) so the next call is more likely to hit the cache.^[9]

By default dynamic=None, automatic dynamic shapes: the first call specializes on observed shapes, and if subsequent calls violate that guard the function is recompiled with symbolic shape support, using a ShapeEnv and SymPy expressions to reason about size relationships.^[10] Setting dynamic=True skips the first specialization and traces symbolically from the start. Setting TORCH_LOGS=recompiles causes the compiler to log each recompilation along with the guards that triggered it, which is the primary debugging tool for guard failures.^[10]

AOTAutograd (joint forward/backward capture)

To accelerate training and not just inference, PyTorch 2.0 introduced AOTAutograd, an "ahead-of-time" autograd engine that traces both the forward and backward graphs by re-using the standard dispatcher.^[1][3] AOTAutograd intercepts the captured Dynamo graph, decomposes operators against the PyTorch dispatcher, runs the autograd engine ahead of time on FakeTensors, and emits a joint forward-and-backward FX graph that the downstream compiler can optimize as a single unit.^[3] Without AOTAutograd, only the forward pass would be compiled, leaving training largely on the eager path; with it, the entire training step (including activation save/restore) becomes a single compiled artifact.^[1]

PrimTorch (operator decomposition)

PyTorch has on the order of two thousand operators (counting overloads), too many for any single backend to implement directly. PrimTorch defines two smaller, stable operator sets that the captured graph is lowered into: the "Prim ops" set of roughly 250 low-level primitives suited for compilers, and the "ATen ops" set of approximately 750 higher-level operators suited for backends that prefer to consume larger building blocks.^[1] Decompositions are written in Python so they can be reused across backends and traced through Dynamo.^[1][3]

TorchInductor (default code generator)

TorchInductor is the default backend that consumes the post-AOTAutograd, post-PrimTorch graph and produces device code.^[1][3] On NVIDIA GPUs it lowers to OpenAI Triton kernels; on CPUs it generates C++ source files compiled with a normal host compiler and parallelized with OpenMP.^[3] Inductor uses a Python loop-level intermediate representation with about 50 operators that is deliberately small and extensible, and it performs scheduling, fusion, and memory-planning passes before emitting kernels.^[1] The ASPLOS 2024 paper reports that Inductor outperforms six other compilers tested in the same harness across 180+ models.^[3]

Code generation proceeds in three logical phases. First, the captured FX graph is decomposed and normalized using PrimTorch decompositions; second, Inductor lowers the result into its loop-level IR and performs scheduling decisions (op fusion, tiling, memory layout); third, it emits Triton kernels (for CUDA and ROCm targets) or C++ source files (for CPU targets) and compiles them with the corresponding toolchain. The Triton path benefits from Triton's block-level programming model, which lets a single kernel author cover a wide range of tile sizes and broadcasting patterns without writing per-shape variants.^[1][3]

Compile modes

torch.compile accepts a mode argument that selects an optimization preset:

Scroll sideways for more →

Additional keyword arguments include fullgraph=True (raise on any graph break, useful for export and debugging) and dynamic=True/False/None (control symbolic-shape behavior).^[9][10] The reduce-overhead mode is implemented via "CUDAGraph Trees", a refinement that records multiple captured graphs sharing a single memory pool so that different execution paths can be replayed without separate allocations.^[12]

CUDA Graph Trees specifically address a problem that arises when chaining several captured CUDA graphs together: a naive approach would force each graph to use a separate memory pool, blowing up activation memory and adding host-side copies to move intermediates between graphs. Trees instead share a single pool across all graphs in the tree and use a tensor-liveness tracker (implemented in torch/_inductor/cudagraph_trees.py) to reuse dead memory across replays, retaining the launch-overhead savings without paying the memory cost.^[12][22] Because CUDA Graphs require fixed tensor addresses and shapes, Trees re-record a fresh CUDAGraph for each unique input shape; for workloads with very high shape variability, the re-record cost can outweigh the savings, in which case mode="default" (without CUDA Graphs) is preferred.^[12][22]

torch.export and AOTInductor

torch.export is a sibling API that reuses Dynamo to capture a "complete" graph (one with no graph breaks) of an nn.Module plus example inputs, then serializes it as an ExportedProgram. PyTorch 2.2 added AOTInductor, an ahead-of-time variant of TorchInductor that consumes an exported program and emits a compiled shared library suitable for loading in non-Python server-side environments, sharing the same Triton/C++ codegen as the JIT path.^[19] This lets a single backend serve both interactive development (via torch.compile) and production deployment (via torch.export and AOTInductor).

Implementations and Backends

Although TorchInductor is the default, torch.compile exposes a backend= argument that selects an alternative compiler; the canonical list can be retrieved with torch.compiler.list_backends() and includes inductor, cudagraphs, onnxrt, openxla, openxla_eval, and tvm, with other backends moved out of core into companion packages.^[14] Custom backends can be registered through the torch._dynamo extension API; the developer documentation walks through the contract a backend must satisfy (it receives an FX GraphModule and a list of example inputs, and returns a callable).^[15]

Notable third-party backends include:

Scroll sideways for more →

The PyTorch project moved most of these backends out of torch core into separate packages over the course of 2023 and 2024, leaving the in-tree set focused on Inductor, CUDA Graphs, and a small number of cross-vendor adapters.^[14]

Performance Numbers

Two distinct sets of headline numbers are commonly cited.

The PyTorch Conference 2022 keynote (December 2, 2022) and the official PyTorch 2.0 announcement reported, across 163 diverse open-source models tested on an A100:^[1]

Scroll sideways for more →

torch.compile was reported to work on 93% of the 163 models tested, with the weighted average defined as 0.75 times the AMP speedup plus 0.25 times the FP32 speedup to reflect that automatic mixed precision is the more common training configuration.^[1]

The ASPLOS 2024 paper, which uses a refreshed harness covering 180+ models on the same A100 hardware, reports a geometric-mean inference speedup of 2.27x and a geometric-mean training speedup of 1.41x for TorchInductor over eager mode, and finds that TorchInductor outperforms each of six other compared compilers across the harness.^[3] The paper also evaluates Dynamo's graph-capture coverage and overhead and presents detailed ablations of guard handling, dynamic shapes, and operator-decomposition policies.^[3]

PyTorch maintains a public nightly performance dashboard that runs the same HuggingFace, TIMM, and TorchBench harnesses on 12 GCP A100 nodes (each with a 40 GB A100) and publishes speedup, memory, and pass-rate metrics for every nightly build.^[17]

A second category of results comes from end-application studies. The November 2023 PyTorch blog "PyTorch compile to speed up inference on Llama 2" reports the following inference latencies, measured on A100 80 GB hardware and with torch.compile plus SDPA (and tensor parallelism for the 70B configuration) compared against unoptimized eager execution:^[20]

Scroll sideways for more →

For inference of compute-bound diffusion and transformer pipelines, the Hugging Face Diffusers PyTorch 2.0 documentation reports that torch.compile can provide "an additional speed-up of 5-300x on top of SDPA" depending on GPU architecture, with Ampere (A100, 3090), Ada (4090), and Hopper (H100) showing the largest gains; the upper end of that range corresponds to small batch sizes where kernel-launch and Python overheads dominate.^[21]

Applications

torch.compile is now the recommended path for accelerating training and inference of standard PyTorch models without rewriting them. Concrete deployment patterns include:

• Speeding up Transformer training in Hugging Face Transformers and similar libraries by adding a single model = torch.compile(model) line after model construction. The PyTorch 2.0 announcement cites Sylvain Gugger reporting a 1.5x to 2x speedup on Hugging Face training scripts with no other changes.^[1]
• Accelerating computer-vision training: TIMM models pass through torch.compile largely without modification, per Ross Wightman's quoted endorsement in the PyTorch 2.0 announcement.^[1]
• Reducing inference latency on small models by combining mode="reduce-overhead" with CUDA Graphs to amortize kernel-launch and Python-interpreter overhead, which dominates eager-mode execution for short generation steps in language models.^[11][12]
• Exporting models to other runtimes via alternate backends, for example backend="onnxrt" for ONNX Runtime or backend="openxla" for XLA-targeted hardware.^[14]
• Distributed training: torch.compile is integrated with both DistributedDataParallel and FullyShardedDataParallel; the announcement reports up to 15% (FP32) and 80% (AMP) gains for compiled DDP.^[1]
• Large-language-model inference: a November 2023 PyTorch blog post from an IBM Research and Meta team reports that combining torch.compile, SDPA (with FlashAttention), and tensor parallelism yields 29 ms/token latency on a 70-billion-parameter Llama 2 model running on 8 A100 80 GB GPUs, a 2.4x improvement over unoptimized inference under the same conditions (512-token input, 50-token generation).^[20]
• High-throughput attention: from PyTorch 2.2 onward, scaled_dot_product_attention automatically dispatches to FlashAttention-2 on supported hardware, achieving 50 to 73 percent of theoretical maximum FLOPs on A100; combining SDPA with torch.compile is the standard recipe in current PyTorch examples.^[19]

It also unlocks ahead-of-time export. The torch.export API leverages Dynamo's graph capture to produce a serializable program representation, and AOTInductor (introduced in PyTorch 2.2) takes that exported graph and compiles it into a shared library suitable for non-Python server-side deployments, sharing backend code with the JIT path.^[19]

Limitations and Criticisms

Despite the speedups, the system has known constraints and rough edges:

• Coverage is high but not 100%. The original announcement places coverage at 93% of 163 tested models, and graph breaks (where Dynamo cannot proceed and falls back to the interpreter) reduce the size of the compiled region; in extreme cases, performance can match or trail eager mode.^[1][9]
• Cold-start compile times are non-trivial, particularly for mode="max-autotune", which searches over Triton kernel variants and can take significantly longer to first-call than mode="default". GitHub issues report that max-autotune can lead to overall execution times longer than eager when models are run few enough times that compilation does not amortize.^[13]
• Guard misses cause recompilation. Programs that pass tensors of many different shapes (variable-length sequences, ragged batches) may recompile repeatedly unless dynamic=True is enabled or shape ranges are constrained. Each recompilation adds latency and produces a distinct cache entry.^[9][10]
• Hardware coverage at launch was narrow. The PyTorch 2.0 announcement explicitly noted that only NVIDIA Volta and Ampere GPUs and CPUs were supported out of the box, with desktop GPUs (NVIDIA 3090 and older) showing lower speedups than A100; broader GPU and accelerator coverage arrived in subsequent 2.x releases.^[1]
• Debugging is harder than eager. Stack traces span Dynamo-rewritten bytecode, AOTAutograd-generated graphs, and Triton-emitted code, requiring developers to learn new logs (TORCH_LOGS=recompiles,graph_breaks,dynamo) and tools (torch._dynamo.explain) to localize regressions.^[9][10]
• max-autotune on CPU is less mature than on CUDA, with only a subset of operations using templated kernels at launch and ongoing work to extend the GEMM template coverage.^[13]

Version Timeline

Subsequent PyTorch 2.x releases have continued to expand torch.compile rather than supersede it. Notable per-release changes include:

Scroll sideways for more →

The PyTorch Foundation publishes per-release notes via the project blog and the GitHub release tags, and the developer mailing list dev-discuss.pytorch.org tracks ongoing torch.compile RFCs.^[4][19]

Comparison

torch.compile is one of several frameworks-level graph compilers active in the deep-learning ecosystem.

Scroll sideways for more →

Compared to TorchScript, torch.compile distinguishes itself by not requiring users to make their models scriptable: Dynamo's bytecode-level capture handles native Python control flow that TorchScript could not. Compared to JAX's jit, PyTorch keeps eager mode as the default and treats compile as an optional wrapper rather than the dominant API.^[1][3]

A second axis of comparison is the codegen target. Where XLA and TVM use their own internal intermediate representations and target a wide range of hardware through compiler back-ends, TorchInductor stands out by emitting source code in two well-known languages: Triton (a Python-embedded DSL designed at OpenAI for GPU kernel programming) and standard C++ with OpenMP. The Python-implemented Inductor IR is intentionally small (around 50 operators), which the ASPLOS 2024 paper argues lets compiler engineers prototype optimizations in Python and avoids the upfront engineering cost of a custom IR per backend.^[1][3]

Related Work

• PyTorch, the framework that hosts torch.compile.
• Triton (compiler), the GPU kernel language TorchInductor emits.
• JAX, a contemporary framework with a different graph-capture model (jit).
• TensorFlow and its XLA backend, the major prior-generation graph compiler.
• ONNX and the onnxrt backend for cross-framework deployment.
• OpenVINO and Intel Extension for PyTorch, an Intel-targeted backend lineage.
• NVIDIA A100, the GPU on which the headline benchmarks were collected.
• Python PEP 523, the CPython frame evaluation API TorchDynamo uses.
• FX, the PyTorch IR that TorchDynamo emits and TorchInductor consumes.

See also

• PyTorch
• PyTorch Lightning
• Triton (compiler)
• XLA (Accelerated Linear Algebra)
• JAX
• TensorFlow
• ONNX
• OpenVINO
• NVIDIA A100
• CUDA
• Hugging Face Transformers
• Python (programming language)
• Deep Learning
• Meta AI (Company)

References

1. PyTorch Team, "PyTorch 2.x: faster, more pythonic and dynamic as ever", PyTorch, 2022-12-02 (updated for the 2.0 release). https://pytorch.org/get-started/pytorch-2-x/. Accessed 2026-05-21. ↩
2. PyTorch Foundation, "PyTorch 2.0: Our next generation release that is faster, more Pythonic and Dynamic as ever", PyTorch Blog, 2023-03-15. https://docs.pytorch.org/blog/pytorch-2.0-release/. Accessed 2026-05-21. ↩
3. Jason Ansel, Edward Yang, Horace He, et al., "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation", Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '24), 2024-04-27. https://docs.pytorch.org/assets/pytorch2-2.pdf. Accessed 2026-05-21. ↩
4. PyTorch Team, "PyTorch 2 paper and tutorial @ ASPLOS 2024", PyTorch Blog, 2024-04. https://pytorch.org/blog/pytorch-pytorch-2-paper-tutorial/. Accessed 2026-05-21. ↩
5. Jason Ansel, "About", jasonansel.com, accessed 2026. https://jasonansel.com/. Accessed 2026-05-21. ↩
6. PyTorch (official account), "We just introduced PyTorch 2.0 at the #PyTorchConference, introducing torch.compile! Available in the nightlies today, stable release Early March 2023", X (Twitter), 2022-12-02. https://x.com/PyTorch/status/1598708792598069249. Accessed 2026-05-21. ↩
7. Brett Cannon, "PEP 523: Adding a frame evaluation API to CPython", Python Software Foundation, 2016. https://peps.python.org/pep-0523/. Accessed 2026-05-21. ↩
8. PyTorch contributors, "torchdynamo (project README)", PyPI, 2022. https://pypi.org/project/torchdynamo/. Accessed 2026-05-21. ↩
9. PyTorch Team, "Dynamo Overview", PyTorch documentation. https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_overview.html. Accessed 2026-05-21. ↩
10. PyTorch Team, "Dynamic Shapes Core Concepts", PyTorch documentation. https://docs.pytorch.org/docs/main/user_guide/torch_compiler/compile/dynamic_shapes_core_concepts.html. Accessed 2026-05-21. ↩
11. PyTorch Team, "Introduction to torch.compile", PyTorch Tutorials. https://docs.pytorch.org/tutorials/intermediate/torch_compile_tutorial.html. Accessed 2026-05-21. ↩
12. PyTorch Team, "CUDAGraph Trees", PyTorch documentation. https://docs.pytorch.org/docs/stable/user_guide/torch_compiler/torch.compiler_cudagraph_trees.html. Accessed 2026-05-21. ↩
13. PyTorch Team, "Using Max-Autotune Compilation on CPU for Better Performance", PyTorch Tutorials. https://docs.pytorch.org/tutorials/unstable/max_autotune_on_CPU_tutorial.html. Accessed 2026-05-21. ↩
14. PyTorch contributors, "[RFC]: Moving most torch.compile backends out of core", pytorch/pytorch issue #109687, GitHub, 2023-09. https://github.com/pytorch/pytorch/issues/109687. Accessed 2026-05-21. ↩
15. PyTorch Forums, "Where to begin developing custom backend for torch.compiler?", discuss.pytorch.org. https://discuss.pytorch.org/t/where-to-begin-developing-custom-backend-for-torch-compiler/191182. Accessed 2026-05-21. ↩
16. Yaoyao Ding, Bojian Zheng, Allan Lin, et al., "Introducing Hidet: A Deep Learning Compiler for Efficient Model Serving", PyTorch Blog, 2023-04-28. https://pytorch.org/blog/introducing-hidet/. Accessed 2026-05-21. ↩
17. PyTorch Team, "PyTorch 2.0 Performance Dashboard", PyTorch documentation. https://docs.pytorch.org/docs/stable/torch.compiler_performance_dashboard.html. Accessed 2026-05-21. ↩
18. Anthony Alford, "PyTorch 2.1 Release Supports Automatic Dynamic Shape Support and Distributed Training Enhancements", InfoQ, 2023-10. https://www.infoq.com/news/2023/10/pytorch21-at-pytorch-con-2023/. Accessed 2026-05-21. ↩
19. PyTorch Team, "PyTorch 2.2: FlashAttention-v2 integration, AOTInductor", PyTorch Blog, 2024-01-30. https://pytorch.org/blog/pytorch2-2/. Accessed 2026-05-21. ↩
20. Antoni Viros i Martin, Brian Vaughan, Davis Wertheimer, Joshua Rosenkranz, Mudhakar Srivatsa, Nelson Mimura Gonzalez, Raghu Ganti, Supriyo Chakraborty, Zhuoran Liu, Geeta Chauhan, Hamid Shojanazeri, "PyTorch compile to speed up inference on Llama 2", PyTorch Blog, 2023-11-07. https://pytorch.org/blog/pytorch-compile-to-speed-up-inference/. Accessed 2026-05-21. ↩
21. Hugging Face, "Accelerated PyTorch 2.0 support in Diffusers", Diffusers documentation, 2023. https://huggingface.co/docs/diffusers/v0.14.0/optimization/torch2.0. Accessed 2026-05-21. ↩
22. PyTorch contributors, "torch/_inductor/cudagraph_trees.py", pytorch/pytorch source tree, GitHub. https://github.com/pytorch/pytorch/blob/main/torch/_inductor/cudagraph_trees.py. Accessed 2026-05-21. ↩