Loading…
Glossary term
A sequence model architecture that tracks a hidden state over time, offered as an alternative to attention.
A state space model keeps a hidden state that is updated as each element of a sequence arrives, much like an RNN, but with a structure that can also be computed in parallel during training.
Models such as Mamba use this to process very long sequences with cost that grows linearly rather than quadratically.
The core recurrence
h = 0.0
A, B, C = 0.9, 1.0, 0.5
outputs = []
for x in [1, 0, 0, 2]:
h = A * h + B * x # update state
outputs.append(C * h) # read outState space models are a leading alternative to attention for long sequences.