Glossary term

A/B testing

Comparing two versions of a system by showing each to different users and measuring the results.

How it works

In an A/B test, users are randomly split into two groups, one seeing the current version (A) and the other the new one (B). Comparing a chosen metric, such as clicks or satisfaction, shows whether the change really helped.

Enough users and a proper statistical test are needed to avoid being fooled by chance.

A/B test

Group A

Current model
Half the users

Group B

New model
Other half of the users

In code

Randomly assign users to groups

import hashlib

def group(user_id):
    h = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
    return "B" if h % 2 else "A"

Why it matters

It is the standard way to confirm that a new model improves real outcomes, not just benchmark scores.

Related terms

Evaluation Model monitoring MLOps