Learn · Intermediate

Gradient accumulation: training with a large batch when it will not fit

Gradient accumulation is a training technique that sums gradients from several small microbatches before changing a model’s weights. It matters because the batch size that produces stable optimization or good hardware utilization is often larger than the batch that fits in accelerator memory.

Key facts

• A microbatch runs forward and backward normally, but its optimizer step is delayed.
• Four microbatches of eight examples can approximate one update from a batch of 32 examples.
• The loss must normally be divided by the number of accumulation steps so the update scale remains comparable.
• Gradient accumulation reduces peak activation memory; it does not make the model weights or optimizer state smaller.

A neural network learns by computing a gradient: a direction that says how each parameter should change to reduce loss on a batch of examples. In ordinary minibatch training, the process is simple: load 32 examples, calculate their average loss and gradient, then ask an optimizer such as AdamW to update the weights once. The catch is memory. During backpropagation, the system must retain activations—the intermediate values used to calculate derivatives—for every example in the current batch. For a long-context language model or a high-resolution vision model, those activations may consume more memory than the weights themselves.

Gradient accumulation changes the schedule, not the learning objective. Suppose the desired batch is 32, but memory only permits eight examples. Run eight examples, compute gradients, and keep them. Do it again three more times. After four microbatches, average the accumulated gradients and perform one optimizer update. The optimizer sees one update based on 32 examples, while the GPU never needed to retain activations for more than eight at once. It is like asking four small groups to each write their suggested edit to a document, combining the edits, and changing the document only after all four groups have reported.

The detail people most often miss is scaling. Many training libraries return a mean loss for each microbatch. If that loss is backpropagated four times without adjustment, the stored gradient is four times larger than the gradient of the equivalent 32-example mean loss. The usual recipe divides each microbatch loss by the accumulation count before backward(), then takes one optimizer step after all microbatches. Some frameworks hide this, so a practitioner should check whether the trainer treats the configured batch size as microbatch, global batch, or per-device batch.

Why aim for a larger effective batch at all? Averaging gradients over more examples reduces noise. That can make an update more stable and can improve accelerator efficiency, but it changes the optimization regime. The large-minibatch work by Priya Goyal and colleagues, Accurate, Large Minibatch SGD, showed that successful scaling requires learning-rate and warmup choices rather than simply multiplying batch size. Nitish Shirish Keskar and colleagues’ large-batch study helped popularize the warning that extremely large batches can converge to different, sometimes less generalizable solutions. Accumulation inherits those tradeoffs because the optimizer update frequency is still the effective-batch frequency.

It is not always identical to a physical large batch. Dropout samples new randomness on each microbatch, which is normally fine because examples in a physical batch would have independent dropout masks too. But Batch Normalization computes statistics from the microbatch, not the effective batch; that can noticeably change behavior. Sequence packing, dynamic loss scaling, gradient clipping, data augmentation, distributed all-reduce order and nondeterministic kernels can also introduce differences. With Adam-like optimizers, the moment estimates update once per accumulated batch, which is usually intended; updating them after every microbatch would be a different algorithm.

Accumulation also does not solve every memory issue. It lowers the activations associated with batch size, but model weights, optimizer states and a transformer’s context-length costs remain. That is why it commonly appears alongside mixed-precision training, gradient checkpointing, sharding and sequence-length reduction. In distributed training, the effective global batch is microbatch size times accumulation steps times number of workers, so changing the worker count silently changes the optimization setup unless another value is adjusted.

The operational tradeoff is throughput. A large physical batch can exploit more parallel work in one pass. Accumulation serializes microbatches, adds no extra examples per optimizer update, and can therefore slow wall-clock training. Its virtue is feasibility and controlled update frequency. For fine-tuning a model on a single GPU, it is often the difference between an out-of-memory crash and a usable run.

Use it deliberately: choose an effective batch based on validation behavior and learning-rate schedule, select a microbatch that safely fits, set accumulation to reach the target, and log all three values. If results change after a hardware migration, compare the global effective batch and number of optimizer steps before attributing the difference to the model. Gradient accumulation is a scheduling trick, but it changes enough of the training system that it deserves to be treated as a first-class hyperparameter.

Ground Truth. (2026, September 16). Gradient accumulation: training with a large batch when it will not fit. Ground Truth. https://groundtruth.day/learn/gradient-accumulation.html

@misc{groundtruth:gradient-accumulation,
  title  = {Gradient accumulation: training with a large batch when it will not fit},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/gradient-accumulation.html}
}

Topics: training · optimization · memory · batch-size