What is the difference between an eval and a benchmark?

An eval is a test you run to see how an AI system behaves. A benchmark is a defined test that multiple people or teams can run as a common point of reference. A benchmark can be used in an eval, but an eval does not need to use a benchmark.

The distinction is useful because the two answer different questions. An eval often asks, “Does this system work well enough for our task?” A benchmark often asks, “How does this system perform on this shared test?”

A concrete example

Imagine you are building an assistant that answers customer questions about your product. You collect 50 questions from support tickets, write down what a correct answer needs to say, and test the assistant after each change. That is an eval. It is tailored to your product, and its results help you decide whether your assistant is improving.

You also run a public question-answering benchmark on the same model. Its tasks and scoring rules are available to others, so its result gives you a broader comparison. That is a benchmark run. It may help you choose a model, but it cannot tell you whether the assistant knows your refund policy.

How they differ

• Purpose: An eval answers a question about a system's behavior. A benchmark provides a common test for comparison.
• Examples: An eval often uses your own user tasks or known failures. A benchmark usually has a published set of tasks.
• Scoring: You choose an eval's scoring method for the decision at hand. A benchmark defines its scoring method as part of its protocol.
• Best use: An eval checks and improves a specific application. A benchmark compares performance on a shared capability test.

These are tendencies, not strict rules. You can run a published benchmark inside your own evaluation workflow. You can also share an eval you created, along with its tasks and scoring rules, so others can run it as a benchmark.

Which should you use?

Use examples from your own application when you need to know whether a change helps your users. Include common tasks and failures you cannot afford to repeat. Run those examples regularly, and inspect individual answers as well as the summary score.

Use a benchmark when you want a repeatable starting point for comparing models or examining a capability you do not yet have enough examples to test yourself. Choose one whose tasks resemble the work you care about, and follow its instructions and scoring rules if you want to compare your result with others.

The strongest approach is often to use both. A public benchmark can reveal broad strengths and weaknesses; your own eval shows whether those strengths survive in your product. Keep the two scores separate, because they measure different work under different conditions.

For more detail, read what AI evals are, what AI benchmarks are, or explore the benchmark library.