Loading…
Glossary term
When test questions leak into training data, making benchmark scores look better than they should.
Contamination happens when test questions, or text very close to them, appear in the training data. The model can then score well by remembering the answers.
It is hard to rule out for models trained on huge web scrapes.
It is a major reason to treat benchmark scores with caution, and why newer benchmarks keep their questions private.