Batch size

Instead of updating after every single example, training groups examples into batches. The model averages the error over a batch and updates once.

Bigger batches make better use of GPUs and give smoother updates but need more memory. Smaller batches are noisier, which can sometimes help generalization.