In one sentence

Evaluation is measuring how well an AI workflow performs, by comparing its output against known correct answers on real cases.

The simple model

You take a test set of real cases with agreed correct answers, run the workflow against them, and compare. The result is a measurement: how often it was right, where it went wrong, and how bad the wrong cases were.

Evaluation is the difference between "the demo worked" and "we measured it on 100 real cases and here is exactly how it did."

Why you care

AI output is probabilistic, so you cannot know how a workflow performs without measuring it. And a demo actively misleads, because someone chose the cases. Evaluation is how you replace impressions with evidence, before you rely on the workflow and every time you change it.

It also tells you the two things that decide whether to deploy: how often it is wrong, and how costly those wrong cases are. See how to think about the cost of errors.

Measure what matters for your task

Different tasks need different measures. A few common ones, in plain terms:

  • How often the output is correct overall.
  • For a step that flags things: how many of its flags are right, and how many real cases it misses. These are sometimes called precision and recall.
  • For generation: whether the answer is grounded in a real source, and free of invented details. See what is a hallucination.

Pick measures that reflect what a good result actually means for your work, not just an impressive-sounding number.

A small example

A routing workflow looked great in a demo. Evaluated on 200 real messages, it routed 90 percent correctly, but its errors clustered in one important category it almost always got wrong. The overall number was fine; the evaluation revealed a specific, fixable weakness the demo had hidden.

What this does not solve

Evaluation measures performance on the cases you test; it cannot guarantee behavior on unseen cases, and a good score is not permission to stop watching. Pair it with monitoring after launch, because real inputs keep changing.