Loading…
Glossary term
The delay between sending a request to a model and receiving a response.
Latency is the delay between sending a request and getting a response. For a chatbot there are two parts: the time to the first token and the speed at which the rest of the reply streams.
It depends on model size, hardware, prompt length, network distance and how busy the system is.
Measure latency
import time
start = time.perf_counter()
reply = model(prompt)
print(f"{time.perf_counter() - start:.2f} s")People notice delay immediately, so latency strongly shapes whether an AI product feels usable.