Loading…
Glossary term
A model trained to score responses by how well they match human preferences.
A reward model is a network trained to score how good an answer is, using human comparisons of pairs of responses. It stands in for human judgment so the main model can be tuned on millions of examples.
Its flaws matter: the model being tuned may learn to exploit them.
The reward model decides what 'good' means during RLHF, so its quality limits the result.