Benchmarking LLM tasks

A human-centered process for evaluating model output

The most common gap in AI work is the gap between the model producing an answer and anyone establishing whether the answer is correct. People build pipelines, run them at scale, write up results, and never document whether the model was actually performing the task reliably. The output is treated as data because it looks like data. The fact that no one verified it tends not to surface until something downstream goes wrong, by which point the wrong has already happened.

Benchmarking is the practice of closing that gap. It is the structured process of evaluating whether a particular task that an LLM is being asked to perform is being performed reliably. The question benchmarking answers is the only question that ultimately matters in production work: when this model returns an answer, how often is the answer correct?

This article describes how benchmarking works, why it has to be human-centered, what counts as a benchmark dataset, what to do when the task is subjective, how many items to evaluate, and why this work belongs at the front of any AI project rather than at the end. The throughline is straightforward. The model’s confidence in its own output is irrelevant. Confidence has to be established by comparison against ground truth, and ground truth has to come from humans first.

The human-first principle

Before an LLM can be evaluated on a task, the task itself has to be well-defined enough that humans can perform it consistently. This is not a stylistic preference. It is a logical prerequisite. If two trained people, working independently from the same instructions, cannot produce the same answer for a given input, then there is no ground truth for the model to be measured against. The disagreement between humans is not a methodological inconvenience. It is a sign that the task itself is not well defined.

This is the inter-rater reliability test in its strongest form, and it is the gate every LLM benchmark must pass before the benchmark is meaningful. Take a sample of inputs. Have two trained people, each working independently from the same written instructions, produce outputs. Compare them. Where they agree, the task is well-formed for those inputs. Where they disagree, you have learned something important: the instructions are unclear, the categories overlap, the task is genuinely subjective, or the inputs themselves are ambiguous. Each of these problems has to be addressed before the LLM enters the picture. If the humans cannot agree, the model has nothing to be measured against, and any “evaluation” of the model reduces to looking at its output and deciding whether it sounds plausible.

Benchmarking, in this sense, is not really about the model. It is about the task. The benchmark dataset documents what the task is by exhibiting examples of correct outputs. The model is tested against that documentation. Without the documentation, the model is being asked to do something that has not been specified, and the project rests on faith.

The diagram above shows a small benchmark in operation. The leftmost columns hold the independent classifications produced by two human coders. Where they agree, the gold-standard label is obvious. Where they disagree (row 002 in the example), the coders meet, discuss, and produce a resolved label that becomes the gold standard for that document. The next column shows what the LLM produced. The notes column captures whether the model agreed with the gold standard. The match column tallies the result. At the bottom, two summary statistics matter equally: the inter-rater agreement between the humans (which tells you whether the task is well-defined) and the LLM accuracy against the gold standard (which tells you whether the model is performing it reliably).

Building a gold standard

A benchmark dataset, sometimes called a gold standard, is a small labeled corpus where the correct answer for each item is known. The point of the dataset is to give you a fixed reference against which the LLM’s output can be compared. The work of building one is also the work of figuring out what the task actually is, which is part of why it is worth doing carefully.

For a classification task, the procedure is direct. Take a sample of items from your full corpus. Have two trained coders, each working independently from the same written instructions, classify every item in the sample. Compare their classifications. Where they agree, that classification becomes the gold label. Where they disagree, the coders meet, discuss the disagreement, and either reach consensus or refer the item to a third coder. The goal is not to suppress disagreement. It is to surface it, because each disagreement points to a gap or ambiguity in the task definition that needs to be resolved before the model is trusted with it.

The output of this process is two artifacts. The first is the labeled dataset itself, which becomes the benchmark. The second is a refined version of the instructions, which becomes the basis of the prompt sent to the LLM. The instructions you write at the start are almost never the instructions you end with. The act of trying to apply them produces clarification and revision, and the revised instructions are what should drive the prompt.

For extraction tasks (pulling specific fields out of documents), the same logic applies but the artifact is a list of correct extractions per document. For tagging tasks, the artifact is a set of correct tags per document. For summarization or other generative tasks, the gold standard is harder to construct, because two people will rarely produce identical summaries. This is where subjectivity enters the picture and benchmarking becomes more difficult.

The problem of subjective tasks

Some tasks do not have a single correct answer. Two skilled professionals reading the same case note can produce different but equally defensible summaries. Two judges scoring the same essay on a holistic rubric can land on different but reasonable numbers. Two reviewers labeling the “tone” of a passage can disagree in ways that reflect their training and not their carelessness. The output of these tasks is interpretive judgment, not classification or extraction.

Benchmarking subjective tasks is harder, but not impossible. The trick is that you are not benchmarking against a single correct answer. You are benchmarking against the range of acceptable answers that humans produce. If three trained reviewers give summaries that all hit the same key points but use different language, the model’s summary needs to hit the same key points; it does not need to use the same language. The metric becomes coverage of required elements, or scoring against a rubric, rather than exact match.

The harder version of the problem is when a task is fundamentally subjective and the model’s output is being used downstream as if it were objective. This is a real validity concern that benchmarks alone cannot resolve. If a system treats the model’s classification of “tone” or “intent” as if it were a fact, but the underlying construct is one where humans disagree substantially, then the system is reporting one model’s interpretation as ground truth. The fix is not better benchmarking. The fix is honest documentation of what the field actually represents, what its reliability is, and how it should and should not be used.

The practical heuristic carries through every case: if you cannot get two trained humans to agree, do not pretend the LLM has produced a fact. Subjective outputs can still be useful, but they need to be described accurately as what they are.

How many items to evaluate

A common question is how large the benchmark dataset needs to be. The answer depends on three factors: the number of categories or fields involved, the variation present in the corpus, and the precision required in the estimate of accuracy.

For a binary classification with two roughly balanced categories, fifty to one hundred items will give a reasonable first read on the model’s performance. With a sample that size, it is possible to distinguish, say, 70 percent accuracy from 90 percent accuracy with confidence. It is not possible to distinguish 90 percent from 92 percent.

For a multi-category classification, a useful rule of thumb is that each category should be represented by at least twenty items, and ideally fifty. A four-category scheme therefore needs eighty to two hundred items in the benchmark. This matters because performance varies by category. A model might do well on three categories and poorly on the fourth, and the benchmark needs enough examples per category to reveal that.

For extraction tasks, fewer items can suffice if each item contains many fields to extract. A document with twenty extracted fields supplies twenty data points per document, so fifty documents in that case is one thousand evaluations.

For corpora with high internal variation, the benchmark sample needs to capture the variation. A sample drawn only from one segment of the corpus will tell you about the model’s performance on that segment, not on the corpus as a whole. Stratified sampling, where the sample is drawn proportionally from known subgroups, is appropriate when the subgroups are known and meaningful.

The general principle is that the benchmark should be large enough that real changes in the model’s performance show up above the noise of the metric. A benchmark of ten items gives almost no resolution. A benchmark of one thousand items gives fine resolution but takes substantial human effort to build. The right size is usually somewhere in between, and it is reasonable to start small, build the pipeline, and expand the benchmark as the project warrants.

Iterating on the benchmark

Benchmarking is not a single test that the model passes or fails. It is an iterative process that runs alongside the development of the prompt, the pipeline, and any post-processing logic. The first benchmark run almost always reveals something that needs to change: a category definition that is too vague, a prompt instruction the model is misinterpreting, an output format that produces parsing errors, an edge case that was not anticipated. Each fix gets re-tested against the same benchmark, and the score moves up or down accordingly.

The benchmark dataset itself should be held constant across iterations. If the dataset changes between runs, the comparison becomes unreliable; what looks like model improvement might just be the difference between two datasets. Lock the benchmark in at the start, save it to disk, and re-run the model against it after every meaningful change. The accuracy number that comes back tells you whether the change helped, hurt, or had no effect.

This is the point in the workflow where prompt engineering becomes legible. Without a benchmark, prompt revisions are guesswork. With a benchmark, every revision is testable. Change the wording of an instruction, re-run the benchmark, see whether accuracy moved. Decisions about which prompt to deploy in production can rest on data rather than on impressions.

A good practice is to track multiple metrics, not just overall accuracy. Per-category accuracy reveals which categories are hardest. A confusion matrix reveals which categories are getting confused with which others. Time per call tracks practical viability. Cost per call tracks economic viability. The dashboard built for the benchmark is, in effect, the dashboard for the project, and the metrics that matter are the ones that connect to decisions you will actually make.

Front-loading benchmarking

The most consequential point about benchmarking is when in the project it should happen. The default pattern in AI work is to build the pipeline first and benchmark it at the end, after the prompt is written, the loop is implemented, the data is processed, and the results are reported. This is exactly backwards. By the time the pipeline exists, decisions have already been made about category definitions, output schemas, prompt structure, and processing logic that the benchmark might invalidate. Reworking those decisions after the pipeline is built is expensive and demoralizing, and people often resist doing it.

The right order is to write the benchmark first. Take a small sample of the corpus before any pipeline exists. Have humans label it. Inspect the labels and refine the task definition. Write the prompt. Run the prompt against the benchmark. Iterate until the model is performing acceptably on the benchmark. Then, and only then, scale up to the full corpus. The pipeline that runs at scale should be the same pipeline that performed well on the benchmark, not a different one assembled hastily and validated afterward.

This is what front-loading evaluation looks like in practice. The benefit is that the most consequential decisions, the ones that are most expensive to change later, get tested against evidence early. The cost is the discipline of doing the slow work before the fast work, which is unfamiliar in many AI workflows. The slow work pays for itself within a single project.

For high-stakes work, benchmarking is not optional under any circumstances. If an LLM is being used to produce outputs that affect a person’s case, a patient’s care, a client’s outcome, or a research finding, the model’s outputs need to be benchmarked against human ground truth as a matter of professional responsibility. The fact that the model produces fluent text and looks confident is not evidence that it is correct. The benchmark is.

Where this leaves the practitioner

The throughline is simple. The model cannot evaluate itself. The platform cannot evaluate the model. The tool’s marketing materials cannot evaluate the model. Only a benchmark, built carefully against human-validated ground truth, can establish whether a particular task is being performed reliably. Until that benchmark exists, the output is plausible-sounding text, and no one knows whether it is correct.

This is the work that distinguishes serious AI projects from speculative ones. Building a benchmark is not glamorous. It involves writing instructions, sampling documents, asking colleagues to do tedious labeling, computing agreement metrics, and revising the task definition multiple times. The resulting dataset is small. The resulting accuracy number is one number on one task. But that one number is the foundation on which everything else rests. Every meaningful task an LLM is asked to perform deserves one, and critical decisions deserve nothing less.