Glossary term

Preference data

Human or AI judgments of which of two responses is better, used to tune models.

How it works

Preference data records which of two responses a person, or a model acting as a judge, prefers. It trains reward models and methods like DPO.

Collecting it well requires clear guidelines, since annotators can disagree.

A preference dataset

	Prompt	Chosen	Rejected
Example 1	Explain gravity	Clear, accurate answer	Vague answer
Example 2	Write a poem	Creative poem	Off-topic reply

Whose preferences are collected, and how, shapes an assistant's values and style.