Loading…
Glossary term
A neural network that processes sequences by passing information from one step to the next.
An RNN reads a sequence one step at a time, carrying a hidden state forward that summarizes what it has seen so far. That makes it natural for text, speech and time series.
Its weakness is that information fades over long sequences and steps cannot be processed in parallel.
An RNN layer in PyTorch
import torch.nn as nn
rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True)RNNs led sequence modelling before transformers, and the ideas live on in newer state space models.