Glossary term

Inference

Running a trained model to produce outputs, as opposed to training it.

How it works

Inference is using a trained model to produce an output: answering a question, labelling an image or generating text. The weights are fixed and no learning happens.

For language models it runs one token at a time, and cost depends on model size, length of the prompt and reply, and the hardware.

Training vs inference

Training

Weights are updated
Done once, at huge scale
Needs gradients

Inference

Weights are fixed
Done constantly, per request
Optimized for speed and cost

In code

Run a model for inference

import torch

model.eval()                  # switch off training behavior
with torch.no_grad():         # no gradients needed
    output = model(input_batch)

Why it matters

Training is a one-time cost, but inference happens every time anyone uses the model, so it dominates the running cost of AI products.

Related terms

Latency Throughput Model serving