CtrlK
Gradient accumulation is a training technique in deep learning that simulates the effect of training with a large batch size by accumulating gradients over multiple smaller mini-batches (called micro-batches) before performing a single weight update. Instead of computing the gradient on a large batch all at once (which may not fit in GPU memory), the model processes several smaller batches sequentially, sums their gradients, and then updates the weights as if it had processed the full large batch.
This technique is fundamental to modern large language model training, where effective batch sizes of millions of tokens are standard but no single GPU can hold even a fraction of that data in memory at once [1].
In standard training with stochastic gradient descent (or its variants), each training step consists of:
With gradient accumulation, steps 1 and 2 are repeated N times (where N is the accumulation steps) before performing step 3. The process becomes:
The division of the loss by N in each backward pass ensures that the accumulated gradient is mathematically equivalent to the gradient computed on a single batch of size N times the micro-batch size.
Let B be the micro-batch size and N be the number of accumulation steps. The effective batch size is B_eff = N * B.
In standard training with batch size B_eff, the gradient update is:
g = (1 / B_eff) * sum_{i=1}^{B_eff} grad(L_i)
With gradient accumulation, processing N micro-batches of size B:
g = (1 / N) * sum_{k=1}^{N} [ (1 / B) * sum_{j=1}^{B} grad(L_{k,j}) ]
These two expressions are mathematically identical: both compute the average gradient over B_eff = N * B examples. This means that, for most purposes, training with gradient accumulation produces the same result as training with the larger batch size directly [2].
Batch normalization is the primary exception to this mathematical equivalence. Batch normalization computes running statistics (mean and variance) over each mini-batch during training. When using gradient accumulation, these statistics are computed over each micro-batch of size B rather than over the full effective batch of size B_eff.
This means the normalization statistics are noisier, which can lead to slightly different training dynamics and final performance. In practice, this is often not a significant issue because:
| Normalization Type | Affected by Gradient Accumulation? | Explanation |
|---|---|---|
| Batch Normalization | Yes | Statistics computed per micro-batch, not effective batch |
| Layer Normalization | No | Statistics computed per example, batch-size independent |
| Group Normalization | No | Statistics computed per channel group within each example |
| RMSNorm | No | Computed per example; standard in modern LLMs |
Gradient accumulation solves a fundamental tension in deep learning: larger batch sizes generally improve training stability and throughput, but GPU memory is finite.
During training, GPU memory must hold:
For a 7-billion-parameter model in mixed precision (FP16 weights + FP32 optimizer states), the parameters and optimizer states alone consume roughly 56 GB. Activations scale linearly with batch size and sequence length. On a GPU with 80 GB of memory (e.g., an NVIDIA A100), the remaining memory after parameters and optimizer states may only accommodate a micro-batch of 1 to 4 sequences [3].
Gradient accumulation allows this limited micro-batch size to be scaled to an effective batch size of hundreds or thousands of sequences, matching the batch sizes used in published training recipes.
Larger effective batch sizes produce gradient estimates with lower variance, leading to more stable training. This is particularly important for:
| Aspect | Small Effective Batch | Large Effective Batch (via accumulation) |
|---|---|---|
| Gradient noise | High | Low |
| Training stability | Lower | Higher |
| Convergence speed (in updates) | More updates needed | Fewer updates needed |
| Wall-clock time per update | Faster | Slower (sequential micro-batches) |
| Memory usage per GPU | Lower | Same per micro-batch |
| Learning rate sensitivity | More sensitive | More tolerant of higher learning rates |
PyTorch makes gradient accumulation straightforward because its autograd engine accumulates gradients by default. Calling loss.backward() adds the computed gradients to any existing gradients in the .grad attribute of each parameter, rather than replacing them. This is why optimizer.zero_grad() is explicitly called to reset gradients, and why omitting that call (intentionally) enables accumulation [4].
A standard PyTorch training loop with gradient accumulation looks like:
accumulation_steps = 4
optimizer.zero_grad()
for i, (inputs, targets) in enumerate(dataloader):
outputs = model(inputs)
loss = criterion(outputs, targets)
loss = loss / accumulation_steps # Normalize loss
loss.backward() # Accumulate gradients
if (i + 1) % accumulation_steps == 0:
optimizer.step() # Update weights
optimizer.zero_grad() # Reset gradients
Key implementation details:
accumulation_steps before calling backward(). This ensures the accumulated gradient is the average over all micro-batches, matching the behavior of training with the full effective batch size.When combining gradient accumulation with mixed precision training (using torch.cuda.amp), the gradient scaler should only unscale and step after all micro-batches have been accumulated:
scaler = torch.cuda.amp.GradScaler()
for i, (inputs, targets) in enumerate(dataloader):
with torch.cuda.amp.autocast():
outputs = model(inputs)
loss = criterion(outputs, targets) / accumulation_steps
scaler.scale(loss).backward()
if (i + 1) % accumulation_steps == 0:
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad()
DeepSpeed, Microsoft's distributed training library, provides built-in gradient accumulation support through its configuration system. Rather than manually implementing the accumulation loop, users specify the desired behavior in a JSON configuration file [5]:
{
"train_micro_batch_size_per_gpu": 4,
"gradient_accumulation_steps": 8,
"optimizer": {
"type": "AdamW",
"params": {
"lr": 1e-4,
"weight_decay": 0.01
}
}
}
In this configuration, each GPU processes micro-batches of 4, accumulates gradients over 8 steps, resulting in an effective per-GPU batch size of 32. If running on 4 GPUs, the global effective batch size is 128.
DeepSpeed handles the loss normalization, gradient synchronization, and optimizer stepping automatically, reducing the risk of implementation errors.
Gradient accumulation and distributed training are complementary techniques that both increase the effective batch size, but through different mechanisms:
In practice, modern LLM training uses both simultaneously. The total effective batch size is:
B_eff = micro_batch_size * accumulation_steps * num_data_parallel_devices
| Technique | Mechanism | Speed Impact | Memory Impact | Hardware Requirement |
|---|---|---|---|---|
| Gradient accumulation | Sequential micro-batches on same GPU | Slower (sequential processing) | No additional memory | Single GPU sufficient |
| Data parallelism | Parallel micro-batches across GPUs | Faster (parallel processing) | Same per GPU | Multiple GPUs |
| Both combined | Sequential accumulation on each of multiple GPUs | Balanced trade-off | Same per GPU | Multiple GPUs |
When combining gradient accumulation with data parallelism, an important optimization is to synchronize gradients only after the final micro-batch in each accumulation cycle, not after every micro-batch. In standard PyTorch Distributed Data Parallel (DDP), this is achieved using the no_sync() context manager:
for i, (inputs, targets) in enumerate(dataloader):
# Skip gradient sync for intermediate micro-batches
context = model.no_sync() if (i + 1) % accumulation_steps != 0 else nullcontext()
with context:
outputs = model(inputs)
loss = criterion(outputs, targets) / accumulation_steps
loss.backward()
if (i + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()
This optimization reduces the communication overhead by a factor of N (the accumulation steps), since the expensive all-reduce operation across GPUs happens once per effective batch rather than once per micro-batch. DeepSpeed handles this automatically based on the gradient_accumulation_steps configuration [5].
Gradient accumulation is a standard component of virtually every large language model training pipeline. The scale of modern LLM training makes it indispensable:
Models like GPT-4, Llama, and Gemini are pretrained with effective batch sizes measured in millions of tokens. For example, Llama 2's training used a batch size that ramped up to 4 million tokens. With a sequence length of 4096 tokens, this corresponds to roughly 1000 sequences per batch. Distributing this across 2048 GPUs with a micro-batch size of 1 per GPU still leaves a factor of roughly 2 to be covered by gradient accumulation [6].
Many training recipes also use batch size warmup, starting with a smaller effective batch size and increasing it during training. Gradient accumulation provides a clean mechanism for this: simply increase the accumulation steps over time.
Fine-tuning is where gradient accumulation is perhaps most commonly encountered by practitioners, because fine-tuning often happens on single GPUs or small clusters with limited memory:
| Training Phase | Typical Effective Batch Size (tokens) | Common Micro-Batch Size | Accumulation Steps (varies with GPU count) |
|---|---|---|---|
| Pretraining (small models, under 7B) | 500K to 2M | 2 to 8 sequences | 4 to 32 |
| Pretraining (large models, 70B+) | 2M to 8M | 1 to 2 sequences | 8 to 64 |
| Supervised fine-tuning | 32K to 256K | 1 to 4 sequences | 4 to 16 |
| RLHF / DPO | 16K to 128K | 1 to 2 sequences | 8 to 32 |
Beyond the basic accumulation loop, several implementation patterns address common pitfalls and optimize performance in production training systems.
When the number of batches in an epoch is not evenly divisible by the accumulation steps, the last cycle will have fewer micro-batches than expected. This is a subtle but important issue: if the loss is divided by N (the expected accumulation steps) but only M < N micro-batches are processed, the accumulated gradient will be scaled incorrectly.
There are two approaches to handle this:
Most production frameworks (Hugging Face Transformers, PyTorch Lightning) handle this automatically.
Gradient clipping is standard practice in LLM training, typically clipping the global gradient norm to a maximum value (e.g., 1.0). When combined with gradient accumulation, the timing of clipping matters:
| Clipping approach | Correct? | Behavior |
|---|---|---|
| Clip after each micro-batch backward | No | Clips incomplete gradients; final accumulated gradient may be wrong |
| Clip after all micro-batches, before optimizer step | Yes | Clips the full accumulated gradient; mathematically correct |
| Clip within DeepSpeed/FSDP framework | Automatic | Frameworks handle the timing correctly |
The correct approach is to accumulate all gradients first, then clip the accumulated gradient as a single operation before the optimizer step. Clipping intermediate micro-batch gradients changes the mathematical equivalence between accumulation and direct large-batch training.
When using gradient accumulation, metrics like loss and learning rate should be logged per optimizer step (i.e., once per accumulation cycle), not per micro-batch. Logging per micro-batch produces N times as many data points and can obscure the true training dynamics.
Similarly, training progress (current step, progress percentage) should count optimizer steps, not micro-batch iterations. A common bug is to report total micro-batch iterations as "steps," which misrepresents how far training has progressed.
The interaction between gradient accumulation and learning rate scheduling is a frequent source of subtle bugs. The key principle is that the learning rate scheduler should step based on optimizer steps (once per accumulation cycle), not based on micro-batch iterations.
Consider a training run with 1000 optimizer steps and a cosine decay schedule. If the scheduler advances after every micro-batch (instead of every optimizer step), and the accumulation factor is 4, the scheduler will complete its full cycle after only 250 optimizer steps, spending the remaining 750 steps at the minimum learning rate. This effectively truncates the schedule and can significantly hurt performance.
Different frameworks handle this differently:
| Framework | Default LR scheduler behavior | Accumulation-aware? |
|---|---|---|
| PyTorch (manual loop) | User controls when scheduler.step() is called | User must handle correctly |
| Hugging Face Transformers Trainer | Steps scheduler per optimizer step | Yes (automatic) |
| PyTorch Lightning | Steps scheduler per optimizer step | Yes (automatic) |
| DeepSpeed | Steps scheduler per optimizer step | Yes (automatic) |
| FSDP (manual) | User controls timing | User must handle correctly |
When writing a manual training loop with gradient accumulation, the scheduler should be stepped inside the same if (i + 1) % accumulation_steps == 0 block as the optimizer:
if (i + 1) % accumulation_steps == 0:
optimizer.step()
scheduler.step() # Step the scheduler here, not outside
optimizer.zero_grad()
Learning rate warmup interacts cleanly with gradient accumulation as long as the warmup is specified in optimizer steps. For example, "warmup for 500 steps" means 500 optimizer steps (each of which includes N accumulated micro-batches), not 500 micro-batch iterations.
DeepSpeed provides the most comprehensive built-in support for gradient accumulation among distributed training frameworks. Understanding its internals helps avoid common issues.
DeepSpeed's gradient accumulation is configured through three related parameters:
{
"train_batch_size": 128,
"train_micro_batch_size_per_gpu": 4,
"gradient_accumulation_steps": 8
}
Only two of these three parameters need to be specified; DeepSpeed computes the third:
train_batch_size = train_micro_batch_size_per_gpu * gradient_accumulation_steps * num_gpus
If all three are specified and inconsistent, DeepSpeed will raise an error.
DeepSpeed optimizes gradient synchronization during accumulation. Rather than running an all-reduce after every micro-batch (which would synchronize gradients across GPUs N times per optimizer step), DeepSpeed averages gradients locally during each micro-batch step and performs a single all-reduce at the end of the accumulation cycle. This reduces communication overhead by a factor of N.
| Approach | All-reduce operations per optimizer step | Communication overhead |
|---|---|---|
| Naive (sync every micro-batch) | N (one per micro-batch) | High |
| DeepSpeed optimized | 1 (after final micro-batch) | Low |
| PyTorch DDP with no_sync() | 1 (after final micro-batch) | Low |
| FSDP without communication | 0 (sync only at step) | Lowest (but uses more memory) |
DeepSpeed's ZeRO (Zero Redundancy Optimizer) memory optimization interacts with gradient accumulation at each stage:
PyTorch's Fully Sharded Data Parallel (FSDP) offers two modes of gradient accumulation:
no_sync()): FSDP skips the inter-GPU reduction for intermediate micro-batches, accumulating unsharded gradients locally. This uses more memory but eliminates communication for N-1 of the N micro-batches.A known difference between FSDP and DeepSpeed is that FSDP may require a higher learning rate to achieve the same convergence rate as DeepSpeed, particularly when using different communication strategies during accumulation. Practitioners should verify convergence when switching between frameworks.
While gradient accumulation is extremely useful, it has inherent limitations:
Increased wall-clock time. Processing micro-batches sequentially takes longer than processing the full batch in parallel. If the hardware can accommodate a larger batch size, simply increasing the batch size directly is always faster than gradient accumulation. Accumulation should be used when memory, not time, is the binding constraint.
No parallelism benefit. Unlike data parallelism, gradient accumulation does not utilize additional hardware. It trades time for memory, processing more data per update at the cost of longer time per update.
Interaction with learning rate. The optimal learning rate depends on the effective batch size. When using gradient accumulation to increase the effective batch size, the learning rate may need to be adjusted upward following the linear scaling rule or similar heuristics.
Stale statistics. Beyond batch normalization, any technique that relies on batch-level statistics (e.g., some contrastive learning methods that compute similarity within a batch) will see different behavior with gradient accumulation, since each micro-batch operates on a subset of the effective batch.
As of early 2026, gradient accumulation is a standard, universally supported feature across all major deep learning frameworks and training libraries:
gradient_accumulation_steps as a direct argumentpmap for data parallelismThe technique remains essential because GPU memory growth has not kept pace with model size growth. Even the NVIDIA H100 with 80 GB of HBM3 memory cannot hold the activations for large batch sizes during training of models with tens of billions of parameters. As models continue to scale and training recipes demand ever-larger effective batch sizes, gradient accumulation will remain a foundational technique in the practitioner's toolkit [7].