Loading…
Glossary term
A technique that rescales the values inside a layer to keep training stable.
Layer normalization rescales the values inside a layer for each example so they have a steady average and spread. That keeps numbers from drifting too large or small as they pass through many layers. It is applied inside every transformer block.
Layer normalization by hand
import numpy as np
def layer_norm(x, eps=1e-5):
return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)Normalization makes very deep networks much more stable and faster to train.