Many AI projects have a collection of examples. Far fewer have an evaluation set.

The difference is intent. A collection helps a team demonstrate interesting behaviour. An evaluation set defines the behaviour that must remain acceptable when the system changes.

Without that baseline, a new prompt can feel better while becoming worse on the cases that matter. A model upgrade can improve fluency and reduce factual precision. A retrieval change can fix one visible failure while quietly removing useful context elsewhere.

Start with the operational decision

An evaluation set should represent the work the system is expected to perform, not simply the data that was easy to collect.

Useful coverage normally includes:

  • Common, ordinary cases
  • High-consequence cases even when they are rare
  • Ambiguous inputs that require escalation
  • Adversarial or malformed inputs
  • Examples where the correct response is to refuse, defer, or ask for more information
  • Previously observed failures

Each example needs an expected outcome or an explicit scoring method. Exact wording is rarely the right target. The test should capture what makes the result useful, safe, and operationally correct.

A score is not enough

A single aggregate percentage can hide the failures that should block release. The system may perform well overall while failing consistently for a particular document type, user group, or high-risk category.

Release thresholds should therefore include both overall measures and important slices. Some behaviours are better treated as hard guardrails: a critical permission violation should not be averaged away by hundreds of correct summaries.

Evaluation changes the delivery conversation

Once a baseline exists, product and engineering discussions become more concrete.

Instead of asking whether the latest version “feels better”, the team can ask:

  • Which behaviours improved?
  • Which regressed?
  • Are the failures concentrated?
  • Did cost or latency change?
  • Is the result inside the agreed release threshold?

The evaluation set also becomes part of incident learning. A production failure should produce a new test case, so the same class of failure is visible before the next release.

Write it before confidence becomes expensive

Evaluation does not need to begin as a large platform. A small, representative set with clear expected outcomes is more valuable than an elaborate dashboard built around weak examples.

The important step is to make acceptable behaviour explicit before the system earns operational trust.