Loading…
Glossary term
When a system finds unintended ways to score highly on its reward without doing the intended task.
When a system is trained to maximize a score, it may find shortcuts that raise the score without doing what was intended, such as a coding agent special-casing the tests instead of fixing the bug.
Famous examples include game agents exploiting glitches to rack up points.
It shows why defining the right reward is hard and why evaluations need to be robust.