Glossary term

Elo rating

A ranking system based on head-to-head comparisons, used by some model leaderboards.

How it works

Elo was designed for chess. Each player has a rating; after a game the winner takes points from the loser, and the amount depends on how surprising the result was. Beating a much higher-rated opponent earns a lot.

Some AI leaderboards apply it to models using votes from head-to-head comparisons.

Expected win chance rises with rating gap

In code

Elo update

def expected(ra, rb):
    return 1 / (1 + 10 ** ((rb - ra) / 400))

def update(ra, rb, score_a, k=32):   # score_a: 1 win, 0.5 draw, 0 loss
    return ra + k * (score_a - expected(ra, rb))

print(round(update(1500, 1500, 1)))   # 1516

Why it matters

It converts many pairwise judgments into a single ranking.

Related terms

Leaderboard Benchmark Evaluation