Transformer

The transformer, introduced in 2017, processes a whole sequence at once and uses attention to let every token look at every other token. That replaced the step-by-step processing of older recurrent networks and made training far easier to parallelize.

A transformer is a stack of identical blocks, each combining an attention layer and a feed-forward layer, with residual connections and normalization to keep training stable.