What are AI benchmarks? A practical guide to comparing your own models
Choose public benchmarks for your next model decision, interpret scores correctly, and run comparisons in Langfuse without building each integration.
Published
Your next model candidate is ready. Your application evals are already in Langfuse. Before choosing a checkpoint or changing your deployment, you want additional evidence about the capabilities that matter to your product.
Can the candidate reason over long documents? Work with financial tables? Recognize when a reference does not support a claim?
You need a benchmark comparison. You do not need benchmark integration to become another engineering project.
An AI benchmark is a defined test of a particular capability: a set of tasks, a procedure for running them, and rules for scoring the results. For your team, its value is a repeatable comparison between model candidates—not a universal verdict on which model is best.
Why run benchmarks when leaderboards already exist?
A leaderboard can help you identify candidates. It cannot substitute for measuring a configuration it did not test.
A published score for a base model is not a score for your fine-tune. Nor does it establish how your quantized deployment or different inference settings will perform. Evaluation tools such as EleutherAI's LM Evaluation Harness explicitly support testing checkpoints, adapters, and quantized models. [1]
Start with the question your team actually needs to answer:
Does this candidate preserve the capabilities we need while meeting our deployment requirements?
That question gives you a reason to run benchmarks yourself. “We want a dashboard of popular benchmark scores” does not tell you what decision the results will support.
Public benchmarks complement your application evals
Keep the tests built from your own users, documents, and known failures. Langfuse supports turning production traces into dataset items and using datasets to compare changes before shipping them. [2] [3]
Your application evals address product-specific requirements. Public benchmarks add separate checks for selected capabilities under a documented test setup.
For example, you might use a financial-reasoning benchmark to compare how two candidates answer numerical questions, while your application suite checks whether the complete workflow retrieves the right filing, follows your output schema, and meets your users' requirements.
Use public benchmarks to add evidence—not to replace your release criteria.
Choose the capability before choosing the benchmark
Start with one or two tests that address a real uncertainty in your model decision. Add more when you can explain what additional evidence they provide.
For document-heavy AI products, these AllEvals packages offer useful starting points:
- Numerical reasoning over financial text and tables: T²-RAGBench FinQA. Producing a numeric answer with the relevant context supplied. It does not test retrieval.
- Reasoning across long document sets: AA-LCR. Answering questions that require synthesizing evidence from multiple long documents.
- Judging whether evidence supports a claim: AttributionBench. Classifying supplied claims as supported or unsupported. It does not measure attribution of the model's own generated answers.
- Answering focused factual questions without retrieval: SimpleQA Verified. Closed-book factual answers, graded against references. The package reports accuracy, not attempted-answer precision or F1.
Read a few tasks before committing to a full run. Check the required input length, expected output, scoring rules, and package caveats.
A familiar benchmark name is not enough. The actual task must help answer your question.
A benchmark is more than its dataset
Using the same questions is only part of a consistent comparison. A runnable benchmark also needs a defined prompt and grading procedure. Evaluation frameworks expose choices such as few-shot examples, answer extraction, and output processing because these are part of the evaluation configuration. [1]
The dataset defines what you test. Record its version, split, and selected items. Do not compare a full run with a smaller subset as though they were identical.
The protocol defines how you test it. Specify the prompt, available context, permitted tools, and generation settings. Giving one candidate retrieval access and another only the question changes the comparison.
The grader defines what counts as success. Exact matching, numeric tolerance, label accuracy, and LLM judging are different scoring choices. Keep the evaluator configuration fixed when comparing candidates.
Record these details alongside the model version and deployment configuration. Langfuse supports versioned datasets so you can rerun experiments against the same dataset state. [2]
A practical benchmark workflow in Langfuse
1. Name the decision and the baseline
Write the decision before running the suite. For example:
Can our new checkpoint replace the current model without an unacceptable drop in financial-document reasoning?
Identify the current model as the baseline. Decide which failures would block the change, and which tradeoffs you are willing to investigate. Set those expectations before seeing the candidate's score.
2. Import the benchmark setup
AllEvals imports the dataset, saved prompt, and evaluator into your Langfuse project. You run the experiment there, selecting the saved prompt and configuring your model and generation settings. [4]
Check the scoring prerequisites first. Code-based evaluators need a code-execution runtime. LLM-based evaluators need a configured judge model in Langfuse; some packages require a specific model. [4] [5]
The import handles the benchmark-specific setup. It does not remove the need to configure your model or inspect how the benchmark will be graded.
3. Validate a small run, then compare candidates
Start with a small selection of items to inspect the outputs and grading behavior before running the full comparison. [6]
Look for truncated context, malformed answers, failed requests, or a grader that is not running. Treat this as an integration check, not a representative benchmark score.
Once the setup works, run the baseline and candidate on the same selected items with the same evaluation protocol. Keep execution failures visible rather than silently dropping them from the comparison.
4. Review the failures, not just the average
Compare the item-level results. Where did the candidate improve? Which previously correct answers became wrong? Are the differences concentrated in a capability your product depends on?
A small score advantage should prompt inspection, not an automatic deployment decision. Repeat uncertain comparisons where necessary, then review the benchmark findings alongside your application evals and deployment constraints.
The result you want is a decision your team can explain—not just another number in an experiment table.
Know what your score does—and does not—claim
A benchmark score describes the tested configuration. It does not establish that your application is ready for every production case.
It also does not automatically reproduce a published leaderboard result. AllEvals documents package-specific differences, including adapted prompts or judges. [6]
For example, the AA-LCR package uses an adapted equality judge with the destination's configured evaluation model. Its scores may differ from the original publisher's reported results. [5]
Such results can still support an internal comparison when you hold the protocol constant. Label them accurately: report the package, selected items, model configuration, and scoring method rather than implying an official result.
Keep benchmark test items out of your own fine-tuning data, and investigate known training overlap before treating a public score as evidence of generalization.
Run the comparison without rebuilding the integration
When your team already runs experiments in Langfuse, the remaining work should not start with preparing another dataset, reconstructing its prompt, and wiring up its grader.
AllEvals packages those benchmark-specific pieces for your existing project. You choose the coverage, inspect the scoring, and make the model decision. [4]
Start with the capability you need to measure. Import the benchmark. Compare your candidates where you already run your evals.
Explore the benchmark library →