Glossary term

Batch normalization

A technique that normalizes values across a batch of examples to speed up and stabilize training.

How it works

Batch normalization normalizes each feature across the examples in a batch, then lets the network rescale it. It speeds up training and lets higher learning rates be used.

It is common in convolutional networks, whereas transformers more often use layer normalization.

Batch norm vs layer norm

Batch normalization

Normalizes across the batch
Common in CNNs
Depends on batch size

Layer normalization

Normalizes within one example
Common in transformers
Works with any batch size

In code

Batch normalization in PyTorch

import torch.nn as nn

bn = nn.BatchNorm2d(num_features=16)

Why it matters

It was a key technique for training deep image models reliably.

Batch normalization

How it works

In code

Why it matters

Related terms

Batch normalization

How it works

In code

Why it matters

Related terms