Glossary term

Model serving

Running a trained model so applications can send it requests and receive results.

How it works

Serving is running a trained model behind an interface so applications can send it requests and receive results. A serving system handles loading the model, batching requests, scaling across machines and monitoring.

Specialized engines optimize this for large language models.

Serving a model

Step 1App sends request
Step 2Load balancer
Step 3Model server batches requests
Step 4GPU runs the model
Step 5Response

In code

A minimal model API with FastAPI

from fastapi import FastAPI

app = FastAPI()

@app.post("/predict")
def predict(text: str):
    return {"label": model(text)}

Why it matters

A great model is useless without reliable, fast and affordable serving.

Related terms

Inference Latency MLOps