Loading…
Glossary term
A shortcut that adds a layer's input to its output, making very deep networks easier to train.
A residual connection adds a layer's input straight to its output, so the layer only has to learn the change. Information and gradients can flow through the shortcut unchanged.
This simple addition made it possible to train networks with hundreds of layers.
A residual block
def residual_block(x, layer):
return x + layer(x) # the shortcutResidual connections are in almost every modern deep network, including every transformer.