batch-size

Gradient accumulation: training with a large batch when it will not fit Lesson

Gradient accumulation adds gradients from several small forward-and-backward passes before updating parameters, letting a training job approximate a larger batch without storing that larger batch in memory at once.