Glossary term

FLOP

A floating-point operation; a measure of computation, with training compute often counted in total FLOPs.

How it works

A FLOP is one floating-point operation such as an addition or multiplication. Total training compute is counted in FLOPs, and chip speed in FLOPS (per second).

A common estimate of training cost for a transformer is about 6 times the number of parameters times the number of training tokens.

Training compute grows with size and data (illustrative multiples)

Small model1x
10x params, 10x data100x
100x params, 100x data10000x

In code

Rough training compute

params = 70e9      # parameters
tokens = 2e12      # training tokens
flops = 6 * params * tokens
print(f"{flops:.2e} FLOPs")

Why it matters

Compute counted in FLOPs is used to compare models, plan budgets and in some policy thresholds.

Related terms

Scaling laws GPU Training