Loading…
Glossary term
A benchmark of hand-written programming problems used to test code generation.
HumanEval has 164 hand-written Python programming problems. A model writes a function and the result is checked by running unit tests; pass@k is the share solved within k tries.
It measured early code models, but top systems now score very high.
A HumanEval-style check
def candidate(nums): # model-written solution
return sorted(nums)[-1]
assert candidate([3, 9, 4]) == 9 # unit test decides pass or failIt popularized running code to grade code, and shows why harder coding benchmarks were needed.