In one sentence
Latency is how long an AI step takes to produce a result.
The simple model
Some AI steps need to answer while a person waits: a customer chat, an interactive assistant. Others run in the background, where a few extra seconds, or minutes, do not matter: processing overnight invoices, drafting summaries for later review.
Latency is only a problem when it clashes with what the workflow needs. A slow step in a background job is fine. The same speed in a live conversation is not.
Why you care
Latency shapes where and how you can use AI:
- For interactive uses, a slow response frustrates people and gets abandoned. Speed is part of whether the workflow is usable at all.
- For background uses, latency barely matters, which gives you more freedom in how the step is built.
- More thorough approaches, retrieving more, checking more, running more steps, are often slower. There is frequently a trade-off between speed and thoroughness.
Knowing whether your step is interactive or background tells you how much latency you can accept, and therefore how much thoroughness you can afford.
A small example
A team wanted a thorough, multi-step AI check on every incoming order. Run live at the point of order, it was too slow and annoyed staff. Moved to a background step that ran the moment an order arrived and flagged problems in a queue, the same thoroughness became an asset, because nobody was waiting on it.
What this does not solve
Latency is about speed, not correctness. A fast answer can be wrong and a slow one right. Manage latency for usability, but judge quality separately through evaluation. And a background design can absorb latency, but it cannot rescue a step that is simply unreliable.
