PyTorch

PyTorch is an open-source software library for tensor computation and machine learning. It provides multidimensional arrays, automatic differentiation, neural-network building blocks, optimizers, data utilities, distributed execution, compiler interfaces, and deployment-oriented graph export. Most users work through the Python torch package, while a C++ runtime implements much of the tensor, operator, autograd, and parallel-execution machinery.^[1] The library is especially associated with deep learning, but its tensor and differentiation facilities can also support other numerical programs.

PyTorch executes operations eagerly by default: Python code runs normally, tensor operations are dispatched as they are encountered, and autograd records the operations needed for differentiation. This is not an exclusively "dynamic graph" system. Modern PyTorch can also capture regions of a program for just-in-time compilation and can export a constrained, ahead-of-time graph for runtimes that do not execute the original Python.^[6][7] The distinction between eager execution, compilation, and export is central to understanding both its flexibility and its limits.

Development began in 2016 among contributors from the Lua Torch community, and the initial public release followed in January 2017.^[1][2] The project moved to the Linux Foundation in 2022, with business governance assigned to the PyTorch Foundation and technical decisions retained by project maintainers.^[2] As of July 8, 2026, the current stable release was PyTorch 2.13.^[3] That dated version identifies the documentation state used here; individual APIs and accelerator backends can have different stability classifications.^[4]

Scope and history

The name PyTorch can refer narrowly to the core pytorch/pytorch repository and the torch package, or more broadly to libraries and tools built around them. This article uses the narrow meaning unless an adjacent project is named. A domain library, model repository, compiler backend, or foundation-hosted project may have its own release schedule, maintainers, supported platforms, security assumptions, and license. Installing torch does not install the entire PyTorch ecosystem.

PyTorch grew out of experience with Torch, a scientific-computing and machine-learning library whose widely used interface was written in Lua. According to the project's 2022 history, PyTorch development started in 2016 as a collaborative effort involving people from that community, with substantial participation from Meta AI and contributions from other organizations.^[2] The first public release in January 2017 joined an array-oriented Python interface to eager execution, reverse-mode differentiation, and accelerator support. The 2019 PyTorch systems paper framed the design around ordinary Python programs, interoperability with the scientific Python environment, and a performance-oriented C++ core.^[1]

This design differed from the graph-construction interfaces common in several earlier deep-learning frameworks. A model could contain Python branches, loops, classes, logging, data processing, and library calls without first translating the whole program into a separate graph language. That choice made normal Python debugging available and let the graph recorded for differentiation change between executions. It also limited the framework's ability to optimize across operations when it could see only one eager operation at a time.

PyTorch 2 addressed that tradeoff without removing eager execution. Released in 2023, its compiler stack added a path that observes Python execution, extracts compatible tensor computations, and compiles those regions while leaving unsupported behavior in Python.^[7] A separate export path produces a more restricted representation for ahead-of-time transformation and deployment. These additions make a simple historical contrast between "dynamic PyTorch" and "static graph frameworks" misleading. Execution model, graph capture, compiler backend, and deployment representation are separate choices in current systems.

The core project does not prescribe one complete training application architecture. It supplies components that applications may compose or replace. Higher-level projects can add configuration, training-loop organization, experiment management, or domain-specific models. For example, PyTorch Lightning builds training abstractions around PyTorch, but it is not the core framework and should not be used as evidence for what the torch package itself guarantees.

Programming and execution model

The fundamental value type is a multidimensional tensor. A tensor has an element type, shape, stride, layout, device, and associated storage. Multiple tensor views can describe the same storage with different shapes or strides, so an operation that appears to create a differently shaped value may not copy its data. Other operations do allocate new storage. The result depends on the particular API, not merely on whether two values have the same numerical contents.^[5]

Device placement is part of a tensor's state. Operations generally require compatible devices and dtypes, and moving or converting a tensor can copy data or change numerical behavior. PyTorch exposes CPU execution and accelerator-specific backends through a mostly shared tensor interface, but that interface does not imply identical operator coverage, algorithms, precision, or performance. A model that runs on one device may need a different build, dtype choice, fallback, or implementation on another.

PyTorch's Python interface resembles NumPy array programming, and the libraries can exchange data. Some conversions can share memory, while others copy because of device, dtype, layout, ownership, or API semantics. DLPack provides another exchange protocol used by array and accelerator libraries. Shared storage is efficient, but it also means that mutation through one view can affect another. Code should therefore document whether an input is borrowed, copied, detached from autograd, or transferred to another device.

Neural-network state is normally represented by nn.Module. A module can contain parameters, persistent buffers, and child modules, and its forward method defines a computation. Functional operators provide stateless forms of many layers and losses. Optimizers hold update state and modify parameters using computed gradients. Dataset and data-loader interfaces organize sample access, batching, shuffling, worker processes, and device-transfer preparation. These are conventions and reusable interfaces rather than a mandatory closed training loop.

When Python calls a tensor operation, a dispatcher selects an implementation based on properties such as device, layout, dtype, and registered dispatch keys. Many operations ultimately run in C++ or in a device library, so Python can schedule work whose expensive numerical portion executes elsewhere. Accelerator launches can be asynchronous with respect to the host. Measuring a Python call without the required synchronization may therefore measure scheduling rather than completed device execution.

This split between Python control and lower-level data execution is a major source of both usability and complexity. Python can decide what operation to run next, while optimized kernels perform the dense numerical work. For workloads made of large operators, launch and interpreter overhead may be small relative to kernel time. For workloads made of many small operations, Python dispatch, kernel launch, memory traffic, and intermediate allocation can dominate. Compilation attempts to reduce some of those costs by exposing larger regions to transformation and fusion.

PyTorch includes extension mechanisms at several levels. Users can define custom autograd functions, register operators and device implementations, write C++ or accelerator extensions, or provide compiler backends. An extension becomes part of the trusted program. Its shape checks, memory safety, gradients, device behavior, serialization, and compatibility do not become correct merely because it is invoked through a PyTorch API.

In-place tensor methods conventionally end in an underscore. They can reduce allocations in some programs, but mutation is not automatically faster and can interfere with views or values saved for differentiation. PyTorch tracks tensor version counters and raises errors for some unsafe changes to saved values.^[6] Absence of an error does not establish that a mutation is a good performance choice, only that the particular recorded autograd constraint was not violated.

Autograd and training semantics

PyTorch autograd implements automatic differentiation, principally reverse mode for the training pattern in which a scalar loss depends on many parameters. During a forward computation, operations involving tensors that require gradients record a directed graph of Function objects. The backward pass traverses that graph using the chain rule to accumulate derivatives at leaf tensors.^[6] Backpropagation is the neural-network use of this reverse accumulation process.

The recorded graph is recreated on each iteration. If Python control flow takes a different branch, the new execution records a different graph. This behavior explains PyTorch's traditional "define by run" description. It does not mean that autograd differentiates arbitrary Python behavior. Differentiation applies through supported tensor operations and registered gradient formulas. Integer indexing decisions, external I/O, arbitrary object mutation, and operations that detach or convert a value can form boundaries.

Users choose which leaves require gradients. Grad mode records eligible operations, no-grad mode omits them, and inference mode can skip additional tracking work but imposes stronger restrictions on tensors created inside it. detach creates a tensor separated from an existing gradient history. These mechanisms are distinct from Module.train() and Module.eval(), which change the behavior of modules such as dropout and batch normalization. Evaluation mode does not by itself disable gradient recording, and no-grad mode does not by itself put a model into evaluation mode.^[6]

Some backward formulas require forward intermediates. Autograd saves those tensors until they are consumed or the graph is released, which can make activations a major memory cost. Gradient checkpointing, saved-tensor hooks, recomputation, and sharding can change that cost, but each adds its own runtime, storage, or correctness considerations. The relevant question is not only how many parameters a model contains, but which activations, gradients, optimizer states, and temporary workspaces remain live at each point.

Gradients accumulate into parameter .grad fields unless code clears or replaces them. An optimizer then reads gradients and its own state to update parameters. Ordering matters: a typical iteration clears gradients, computes a forward loss, runs backward, and performs an optimizer step, but accumulation across microbatches deliberately changes that sequence. Distributed wrappers can also insert gradient communication during backward. A training result therefore depends on optimizer state, reduction order, batch construction, precision, and synchronization as well as on the forward model.

Mixed-precision training uses lower-precision values or operations to reduce memory and increase throughput on suitable hardware. It can require loss scaling, selective higher-precision accumulation, and checks for overflow or underflow. The same nominal dtype can use different kernels or accumulation modes on different devices. Convergence and numerical tolerance must be validated for the actual model and backend rather than inferred from the presence of mixed-precision support.

Compilation, export, and deployment

torch.compile accepts a Python function or module and asks TorchDynamo to capture compatible regions during execution. TorchDynamo hooks into CPython frame evaluation, interprets Python bytecode symbolically, and constructs FX graphs for tensor operations. It also records guards for assumptions about inputs, tensor metadata, Python values, module state, and other properties. A backend then compiles each captured graph; Inductor is the default backend in PyTorch 2.13.^[7][8]

Compilation is guarded specialization, not a one-time translation of every possible input. If a later call violates a guard, PyTorch may compile another version. If recompilation reaches its configured limit, execution can fall back to eager mode. Changes in shape, dtype, Python constants, module attributes, or control-flow conditions can cause new specializations. Dynamic-shape tracing can reduce some recompilation, but some operations and optimizations still require specialization.^[8]

When TorchDynamo reaches unsupported behavior, it can end a graph, run that portion in ordinary Python, and resume capture afterward. This is a graph break. Graph breaks preserve flexibility, but smaller graph fragments can reduce optimization opportunities and add transitions. With full-graph capture requested, a break is an error instead of a fallback. Debugging compilation therefore includes inspecting breaks, guards, recompilations, generated code, compile time, memory, and numerical results, not just measuring steady-state latency.

The compiler stack separates capture from optimization. The PyTorch 2 systems paper describes AOTAutograd as a component that obtains forward and backward graphs and decomposes operations, while Inductor lowers graphs to loop-level representations and generated code.^[7] Inductor has used Triton for GPU code generation and C++ for CPU code, while current implementations can add or change lowerings and backends. Those implementation choices should not be treated as a permanent list of supported hardware.

Compiler performance is workload-specific. Fusion can remove intermediate memory traffic and launches, but capture and compilation have an upfront cost. A compiled graph can also use more workspace memory, select a kernel poorly for a shape, or encounter an unsupported operator. Comparisons need warmup policy, compile time, input distributions, precision, device, software versions, and an eager baseline. A speedup reported for one model on one accelerator is not a general property of torch.compile.

torch.export serves a different purpose. It takes a module and example inputs and produces an ahead-of-time graph of tensor computation with recorded input constraints. The graph is normalized and does not contain ordinary Python semantics. By default, example dimensions are specialized; dimensions intended to vary must be declared dynamic. Explicit control-flow operators can represent supported dynamic behavior. If the required full graph cannot be captured, export normally reports an error rather than leaving an untraceable region to execute in Python.^[9]

This restriction makes an exported program easier to serialize, transform, and run outside the original Python environment, but the graph remains subject to operator, shape, and backend support. Official documentation now marks TorchScript deprecated and recommends torch.export for new export work.^[10] Existing TorchScript artifacts do not automatically become export programs, and either artifact type should be treated as executable content when it comes from an untrusted source.

Export is one stage of model deployment, not a universal runtime format. The ONNX exporter targets a separate interchange specification and downstream runtimes. AOTInductor can compile an exported program for supported server or C++ settings. ExecuTorch uses PyTorch 2 compiler and export facilities to prepare models for a lightweight on-device inference runtime, rather than relying on the legacy TorchScript-based PyTorch Mobile path.^[17] Each route needs target-specific operator coverage, numerical comparison, memory testing, packaging, and rollback procedures.

Distributed and accelerator execution

PyTorch's distributed training package supplies collective communication, process groups, launch and elasticity utilities, distributed checkpointing, and parallel wrappers. It does not turn an arbitrary local program into an efficient cluster job automatically. Correctness and throughput depend on process placement, input partitioning, collective order, failure handling, network topology, device affinity, and the interaction between computation and communication.

DistributedDataParallel, usually called DDP, implements synchronous data parallelism. Each process holds a model replica and computes gradients from a different portion of data. During backward, corresponding gradients are combined across processes so replicas apply consistent updates. DDP groups gradients into buckets, allowing communication for a completed bucket to overlap with backward computation for earlier layers.^[11] The application still has to partition input data and coordinate optimizer steps.

DDP replicates parameters and usually optimizer state on every participating process. That is straightforward when a complete model fits on each device, but it does not solve model-state memory growth. Communication can also dominate when gradients are large relative to computation or when the interconnect is slow. Bucket order and size, unused parameters, uneven work, collective backend, network hierarchy, and gradient-accumulation strategy can all change results or throughput.

Fully Sharded Data Parallel, or FSDP, reduces replicated state. It divides a model into units and shards parameters, gradients, and optimizer state across ranks. Before a unit computes, ranks communicate to materialize the required unsharded parameters; after computation, parameters can be resharded, and reduce-scatter distributes reduced gradients.^[12] This lowers persistent memory per rank at the cost of more complex materialization, communication, prefetch, and lifetime management.

FSDP is not simply "DDP for larger models." Unit boundaries, resharding policy, mixed precision, checkpoint format, optimizer behavior, and communication topology determine the memory-throughput tradeoff. Some optimizers depend on a complete parameter or global state and need special handling. Shared parameters and composition with other parallel methods can also constrain wrapping. The 2023 FSDP paper describes these limitations as well as the sharding mechanism; its benchmark results are historical measurements, not current cluster guarantees.^[12]

Tensor parallelism partitions individual tensor computations across devices, while pipeline parallelism places different model stages on different devices and schedules microbatches through them. These methods can be combined with data sharding to form multidimensional strategies. They solve different memory and throughput problems and introduce different collective patterns, bubbles, and model-structure requirements.

Accelerator behavior remains backend-specific. A graphics processing unit may execute kernels asynchronously, use reduced-precision units, and depend on external math and communication libraries. CUDA is a major NVIDIA execution backend for PyTorch, but it is neither required for CPU use nor a portable name for every accelerator. A wheel or source build determines which backends and device architectures are present. Operator availability in documentation does not ensure that every dtype, shape, layout, gradient, compiler path, or distributed combination is implemented on every device.

Distributed security is an operational constraint. The PyTorch security policy says that c10d, RPC, and TCPStore are intended for trusted internal networks. It states that these primitives do not add authorization and send messages unencrypted, and that a party with network access can cause work to execute with the privileges of the PyTorch process.^[14] Network isolation, host authentication, least privilege, secret separation, and workload sandboxing must be supplied by the deployment environment.

Reliability, serialization, and security

A checkpoint can contain model parameters, persistent buffers, optimizer state, scheduler state, random-number-generator state, and application metadata. For module compatibility, official guidance recommends saving a state_dict rather than pickling an entire module object.^[13] Loading still requires code that constructs a compatible module, and key names, tensor shapes, dtypes, preprocessing, optimizer definitions, and framework behavior can change across versions.

torch.save and torch.load use Python pickle for general objects. Pickle can invoke code while reconstructing objects. Since PyTorch 2.6, torch.load uses weights_only=True by default when the caller does not supply a custom pickle module. The restricted unpickler permits the limited constructions needed for plain tensor state dictionaries and blocks dynamic imports by default. It narrows remote-code-execution exposure, but the serialization note explicitly says it does not prevent denial of service and may not eliminate memory-corruption risks.^[13]

Loading with weights_only=False, adding classes to an allowlist, importing a model package, or executing custom operators expands the trusted code base. Those choices should be made only after verifying provenance. The core security policy recommends separating model code from weights, checking provenance or cryptographic checksums, and running untrusted models in an isolated environment.^[14] A narrower format such as Safetensors can avoid Python object reconstruction for weight storage, but it cannot validate the intent of model code, stop resource exhaustion, or make downstream kernels memory-safe for every malicious input.

Model artifacts are therefore not passive documents. A repository can include installation scripts, Python modules, custom C++ or accelerator extensions, compiled caches, configuration hooks, and serialized objects. Even an inspection tool may execute part of an artifact. A secure intake process should fetch by immutable identifier, verify checksums and signatures where available, inspect without importing when possible, isolate conversion, restrict network and filesystem access, and publish a separately verified internal artifact.

Reproducibility is also bounded. PyTorch documentation does not guarantee complete reproducibility across releases, commits, platforms, or CPU and GPU, even with identical seeds.^[15] Randomness can arise from PyTorch, Python, NumPy, data-loader workers, libraries, and device kernels. Deterministic-algorithm settings can reject or replace some nondeterministic operations, but deterministic alternatives can be slower and do not erase cross-platform or cross-version differences.

Numerical equality is a separate issue from randomness. Floating-point addition and multiplication are not associative, so different reduction orders can produce different rounded results. A batched operation need not be bitwise identical to a loop over slices, and CPU and accelerator backends can choose different algorithms or accumulation precision.^[16] Reduced precision, fused kernels, compiler transformations, and distributed reductions can change operation ordering while remaining within an accepted numerical tolerance.

Tests should match the risk. Gradient checks, eager-versus-compiled comparisons, reference-device comparisons, finite-value checks, distributional metrics, and task-level acceptance criteria answer different questions. Bitwise equality may be appropriate within a fixed deterministic environment, while relative and absolute tolerances are more appropriate for many floating-point kernels. A tolerance should come from the application and dtype, not be widened until a failing change passes.

For a reproducible training or inference record, capture source revision, dependency lock, PyTorch build, device and driver details, backend libraries, compiler options, precision, determinism settings, seeds, data identifiers, preprocessing, distributed topology, checkpoint hashes, and evaluation code. This metadata does not guarantee identical results, but it makes deviations diagnosable. Production deployment should additionally record input contracts, resource limits, warmup, health checks, monitoring, rollback artifacts, and the trust decision for every loaded component.

Security fixes add a versioning reason to avoid indefinite deployment of an old environment. The project security policy says fixes are applied to the current release and are not backported to older PyTorch releases.^[14] An organization that pins for reproducibility therefore needs a tested upgrade process, rather than assuming that a reproducible old image remains an acceptable security baseline.

Governance, licensing, and ecosystem

PyTorch became a top-level project of the Linux Foundation in September 2022 under the PyTorch Foundation.^[2] The foundation's business governance is distinct from technical governance. Current technical rules describe contributors, module maintainers, core maintainers, and a lead core maintainer. Module maintainers oversee defined areas, while core maintainers set direction and resolve escalations. Technical roles belong to individuals on the basis of contribution and responsibility rather than to companies purchasing seats.^[18]

The core repository is distributed under BSD-style three-clause terms and includes notices for bundled third-party components.^[19] This supports describing the framework source as open-source AI software. It does not determine the license of every dependency, model, dataset, binary component, domain library, extension, or project hosted by the foundation. Users distributing an application must review the licenses and notices for the exact artifacts they ship.

The ecosystem boundary matters when evaluating compatibility and support. A package may call PyTorch APIs while following a different release schedule. A model may be implemented in PyTorch but distributed under a restrictive model license. A hardware extension may depend on private interfaces and break after an upgrade. Conversely, an external project can become a foundation project without becoming part of the torch wheel or inheriting every core guarantee.

Comparisons with TensorFlow or JAX should name the interface, version, workload, hardware, and deployment goal being compared. All three systems have combinations of eager execution, tracing or graph capture, compilation, automatic differentiation, accelerator support, and distribution. The meaningful choice is often between particular programming and operational paths, not between timeless labels attached to entire frameworks.

References

1. ^Adam Paszke et al. "PyTorch: An Imperative Style, High-Performance Deep Learning Library." NeurIPS 2019. Paper
2. ^Soumith Chintala. "PyTorch strengthens its governance by joining the Linux Foundation." PyTorch, September 12, 2022. Announcement
3. ^PyTorch Team. "PyTorch 2.13.0 General Availability." PyTorch Developer Mailing List, July 8, 2026. Announcement
4. ^PyTorch Contributors. "PyTorch documentation." Version 2.13. Documentation
5. ^PyTorch Contributors. "torch.Tensor." PyTorch documentation. Documentation
6. ^PyTorch Contributors. "Autograd mechanics." PyTorch 2.13 documentation. Documentation
7. ^Jason Ansel et al. "PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation." ASPLOS 2024. Paper
8. ^PyTorch Contributors. "torch.compile." PyTorch 2.13 documentation. Documentation
9. ^PyTorch Contributors. "torch.export." PyTorch documentation. Documentation
10. ^PyTorch Contributors. "CPU threading and TorchScript inference." PyTorch 2.13 documentation. Documentation
11. ^Shen Li et al. "PyTorch Distributed: Experiences on Accelerating Data Parallel Training." VLDB 2020. Paper
12. ^Yanli Zhao et al. "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel." PVLDB 16, no. 12, 2023. Paper
13. ^PyTorch Contributors. "Serialization semantics." PyTorch 2.13 documentation. Documentation
14. ^PyTorch Contributors. "Security Policy." `pytorch/pytorch`. Policy
15. ^PyTorch Contributors. "Reproducibility." PyTorch 2.13 documentation. Documentation
16. ^PyTorch Contributors. "Numerical accuracy." PyTorch 2.13 documentation. Documentation
17. ^ExecuTorch Contributors. "ExecuTorch Overview." ExecuTorch documentation. Documentation
18. ^PyTorch Contributors. "PyTorch Governance: Mechanics." PyTorch 2.13 documentation. Governance
19. ^PyTorch Contributors. "License." `pytorch/pytorch`. License