Loading…
Glossary term
A function in a neural network that decides how strongly a unit's output is passed on, letting the network model nonlinear patterns.
An activation function is applied to each unit's output. Without it, stacking layers would still only compute a straight-line relationship; the non-linearity is what lets networks model complex patterns.
Common choices are ReLU (zero for negatives, the input otherwise), sigmoid and GELU.
Common activation functions
import numpy as np
relu = lambda x: np.maximum(0, x)
sigmoid = lambda x: 1 / (1 + np.exp(-x))
print(relu(np.array([-2, 0, 3]))) # [0 0 3]The choice of activation affects how well and how fast a network trains.