You cannot evaluate what you never defined
Why AI projects fail when evaluation is treated as the final step instead of the first one
I spend a lot of time consulting on AI tool development for people building their first system. They are smart, motivated, and domain experts in their fields. They almost always want to start in the same place: the technology. What model should we use? Should we build a chatbot? What platform should it run on?
These are reasonable questions, but they are the wrong first questions to ask. The first question is: how will you know if this thing works?
This is not a philosophical point. It is a project management discipline that, in my experience, separates AI tools that actually get deployed from those that stall out in perpetual prototyping. If you cannot articulate what correct performance looks like before you start building, you will not be able to identify incorrect performance after you finish. You will have a system that produces outputs, and no principled basis for determining whether those outputs are acceptable.
An important distinction before I go further. When you are learning to build AI tools, the natural and pedagogically sound sequence is to build first and then evaluate what you built. This is how you develop intuition for what the technology can and cannot do. You need to see a language model retrieve documents, populate form fields, or draft summaries before you can design meaningful criteria for judging whether it did those things well. Courses and clinics that structure the work this way are doing something right; you cannot evaluate a capability you have never observed.
But the sequence changes when the context changes. When a system moves from a learning exercise to a tool that someone will rely on, a self-represented litigant navigating a court process, a legal aid attorney making triage decisions, a caseworker acting on extracted data, evaluation design moves to the front of the process. Not because the learning sequence was wrong, but because the stakes are different. In a classroom, a system that produces a plausible but incorrect output is a teaching moment. In a legal aid office, it is a missed deadline, an incorrect filing, or a client who does not return. The discipline I am describing in this post is what the work looks like when you are building for deployment, and it is the standard that anyone learning to build AI tools is working toward.
Evaluation is a design activity
Most people think of evaluation as something that happens at the end. You build the system, you run some tests, and you see how it did. This is backwards. Evaluation criteria should be the first deliverable of any AI development project, not the last.
Here is what I mean concretely. Before you write a line of code, before you select a model, before you design an interface, you should produce two documents. The first is a functional specification that defines what the system does and does not do. The second is an evaluation benchmark that defines, for every capability in the specification, how you will test whether the system performs that capability correctly.
These two documents are developed together because they are logically dependent. You cannot write a meaningful test case without knowing what the system is supposed to do. And you cannot claim the specification is complete until you can articulate how each requirement would be verified. If you find yourself writing a specification requirement for which you cannot imagine a concrete test, that is a signal that the requirement is too vague to build against.
The reason this matters for people new to AI development is that language models produce fluent, confident output regardless of whether that output is correct. Unlike traditional software, where a bug typically produces an obvious error or crash, a language model that fails does so by generating plausible-sounding text that is wrong. If you do not have predefined criteria for what “right” looks like, you will not catch it.
The two-person test
I use a heuristic I return to constantly, both in my own work and in consultations: for any task you want the AI to perform, ask whether two trained humans could perform it independently and arrive at the same answer.
This is not a metaphor. It is a practical test you should actually apply. Take the task you want to automate. Write down the inputs. Give those inputs to two qualified people, separately, without letting them communicate. Compare their outputs. If they produce the same result, the task is a strong candidate for AI automation. If they frequently disagree, you have a problem that needs to be solved before any technology enters the picture.
The logic is straightforward. We want AI systems to be fungible with competent human decision-making. That is the standard. An AI that extracts the correct filing deadline from a court order is performing a task that a competent paralegal would also perform correctly, given the same document. An AI that identifies which SCAO form is required for a particular motion type is performing a task that two trained legal aid staff would agree on. These are tasks with verifiable answers, and verifiable answers are what make reliable systems possible.
When the two humans disagree, it means the task involves judgment that is not fully captured by the available inputs, the existing rules, or both. That does not mean the task cannot be automated; it means it requires substantially more work before it can be. You need to define the decision rules more precisely. You need to identify the sources of disagreement and resolve them through clearer criteria, better training, or a narrower scope of tasks. Skipping this step and handing the task to an AI produces a system that is confident and inconsistent; exactly the kind of system that erodes trust.
Why rubric-based evaluation is harder than it looks
This principle has a direct implication for how you evaluate AI systems, and it is the place where I see people get into trouble most quickly. When an AI performs a judgment task, drafting a legal memo, summarizing a client’s situation, or identifying the strongest arguments in a case file, the natural instinct is to evaluate the output with a scoring rubric. Rate the quality of the summary on a 1-to-5 scale. Score the memo’s accuracy, completeness, and organization. This feels rigorous. It produces numbers. Numbers feel like measurement.
The problem is that a rubric creates the appearance of objectivity by attaching numbers to judgments, but if the underlying judgments are subjective, the numbers inherit that subjectivity. And for most AI judgment tasks worth evaluating, the underlying judgments are subjective.
Consider an AI system that drafts a section of a country conditions report for an asylum case. You ask two experienced immigration attorneys to evaluate the output using a rubric. One dimension is “comprehensiveness”: does the report address the relevant country conditions for this particular claim? The rubric defines “5” as “thoroughly addresses all material conditions” and “1” as “substantially incomplete.” Attorney A scores it a 4; Attorney B scores it a 3. Neither is wrong. They weigh the same evidence differently. They have different thresholds for what constitutes “thorough.” The rubric channels their judgment; it does not eliminate their disagreement.
Now consider a different task: the AI extracts the respondent’s date of entry from an immigration filing and populates a form field. You ask two paralegals to check whether the extracted date is correct. They will agree every time, because the task has a verifiable answer. The date is either right or it is not.
The difference between these two evaluation scenarios lies in whether tasks are ready for reliable evaluation or require substantially more groundwork. When humans cannot agree on how to score an AI’s output, the evaluation itself becomes unreliable. You end up with scores that reflect evaluator variability as much as system performance. You cannot distinguish a system that is performing well from one that is performing poorly, because your measuring instrument is imprecise.
This does not mean judgment tasks are off-limits for AI, or that rubrics are useless. It means you need to understand the reliability of your evaluation method before you trust its results. The practical step is to have multiple evaluators independently score the same set of AI outputs, then measure their agreement. If two attorneys applying your rubric to the same outputs agree at a high rate, the rubric is producing a reliable signal. If they frequently disagree, you do not have an AI performance problem; you have an evaluation design problem. The rubric needs sharper criteria, the scoring anchors need concrete examples, or the task needs to be decomposed into smaller components that humans can assess more consistently. Fix the evaluation instrument before drawing conclusions about the system it is supposed to measure.
What counts as “high enough” agreement depends entirely on the task and its consequences, which brings us to a question that every AI project must answer explicitly.
What is an acceptable error?
No system is perfect. This is true of AI systems, and it is equally true of the human performance against which those systems are compared. Humans make mistakes. Some of those mistakes are systematic; a consistent bias in how a particular attorney interprets a particular type of claim. Some are random; an error introduced by fatigue, distraction, or the simple reality that attention is finite. Human error is a baseline fact of every professional workflow, not an aberration.
AI systems also make mistakes, and for different reasons. A language model may misinterpret a query, retrieve the wrong document, or generate a confident response based on a subtle misreading of context. These errors look different from human errors, but they are errors nonetheless. The question is not whether the AI will make mistakes. It will. The question is what error rate is acceptable for the specific task it is performing, and how that rate compares to the alternative.
This is where I see people adopt one of two unrealistic positions. The first is the expectation that the AI must achieve perfect performance before it can be trusted. This standard is not applied to the humans currently performing the same work, and applying it exclusively to AI creates a threshold that no system, human or machine, can meet. A legal aid organization that processes hundreds of intake forms per month does not expect its staff to make zero errors. It expects a manageable error rate, with review processes in place to catch mistakes before they cause harm. Holding an AI tool to a standard of perfection that the existing workflow does not meet is not rigor; it is a reason to never deploy anything.
The second unrealistic position is the assumption that because AI is “just a tool,” any error rate is acceptable as long as a human reviews the output. Human review is essential, but it is not magic. If the AI produces errors at a high rate, the review process becomes the bottleneck, and the efficiency gains that justified the tool disappear. Worse, reviewers develop automation complacency; when most outputs look correct, attention drifts, and errors that a fresh reading would catch slip through. A tool that requires the reviewer to redo the work is not a tool.
The realistic position sits between these extremes, and it requires you to think carefully about two things: the consequences of error and the current baseline.
Some tasks have very low tolerance for error because the consequences are severe and irreversible. An AI system that populates court filing dates has to get them right, because an incorrect date can result in a missed deadline, a default judgment, or a waived right. For tasks like these, the acceptable error rate is effectively zero for the specific fields that carry legal consequences. That does not mean the system is useless if it occasionally makes mistakes elsewhere; it means the high-stakes components need to be evaluated to a higher standard and may require mandatory human verification, regardless of the overall system's accuracy.
Other tasks tolerate more error because the consequences are recoverable or because the task itself involves variable human performance. An AI system that drafts an initial summary of a client’s situation for attorney review does not need to produce a perfect summary every time. It needs to produce a summary that is sufficiently accurate to serve as a useful starting point, and the attorney’s review is a built-in correction step, not an afterthought. For tasks like these, the relevant question is whether the AI’s output saves the attorney enough time to justify the occasional need for correction.
The practical framework is this: for each task your system performs, define the acceptable error rate before you begin evaluation. Ground that definition in two reference points. First, what is the consequence of an error in this task? A wrong filing date has different consequences than a slightly incomplete case summary. Second, what is the current human error rate for this task, if it is known or estimable? If human intake staff currently misroute 8% of cases to the wrong legal category, an AI system that misroutes 5% is an improvement, even though it is imperfect. If human staff currently achieve near-zero error on date extraction, the AI needs to match that standard.
Defining acceptable error forces a conversation that many project teams avoid: what are we actually trying to achieve with this system, and how good does it need to be to achieve it? That conversation is far more productive when it happens during evaluation design than when it happens after someone notices the system got something wrong in practice.
The core is easy; the boundaries are where systems fail
Here is something that consistently surprises people when they start writing evaluation benchmarks: the obvious cases are not the problem. If you build a legal document retrieval system and test it with a query that matches a single statute perfectly, the system will get it right. If you build a form classifier and test it with a filing that is unambiguously one form type, the classifier will get it right. The core of most tasks is straightforward, and AI systems handle the core well.
The work is at the edges.
Edge cases are the inputs that sit at the boundary of your system’s defined capabilities. A query that partially matches three legal provisions but perfectly matches none. A client situation that could require any of several different court forms. A request that is almost but not quite outside the system’s scope. These are the cases where performance degrades, where confident wrong answers appear, and where users lose trust.
What makes edge cases critical for evaluation design is that you cannot identify them after the fact. If you wait until the system is built and then start testing, you will naturally gravitate toward the clean cases; the ones you already have in mind, the ones that confirm the system works. The messy cases, the ones that probe boundaries and expose ambiguity, require deliberate effort to construct. They require you to sit with the specification and ask: where does this get hard? Where would a reasonable query fall between two categories? Where would two domain experts disagree about the correct response?
This connects directly to the two-person test. For the core cases, two trained humans will agree easily. A query asking for the form required to file a motion to set aside a default judgment returns that form; there is nothing to argue about. But consider a self-represented litigant who describes a situation involving both a landlord-tenant dispute and potential housing discrimination. Does the system retrieve resources for a standard eviction defense, for a fair housing complaint, or for both? Should it flag that the situation may require coordinated filings in different forums? Two legal aid attorneys might handle this intake differently, and that disagreement is precisely the information you need before you build the system, not after a self-represented litigant encounters it without any attorney present.
The practical discipline is this: for every capability in your specification, write test cases at three levels. First, the clear cases where the correct answer is obvious. Second, the boundary cases where reasonable people might disagree about the correct response. Third, the cases that should be declined or redirected because they fall outside the system’s scope. The boundary cases are the hardest to write and the most valuable. They force you to make decisions about your system’s behavior in the gray areas; decisions that will otherwise be made implicitly by whatever the model happens to generate on a given day.
If you skip this step, you will end up with a system that performs impressively in demonstrations and unpredictably in practice. Demos use clean inputs. Real users do not.
Every task in the pipeline needs verification
A further practical point that people new to AI development frequently overlook: a multi-step system is only as reliable as its least reliable step, and errors early in the pipeline compound.
Consider a legal document automation system that first identifies the type of court filing required, then extracts relevant facts from the client’s intake narrative, then populates the appropriate form fields. If the first step misidentifies the filing type, the extraction step will look for the wrong facts, and the completed form will be internally consistent but wrong for the client’s situation. Each subsequent step performs correctly on its own terms; given its inputs, it does what it was designed to do. But the inputs were corrupted upstream, and the downstream steps have no way to detect that. A self-represented litigant receives a professionally formatted document that files the wrong motion.
This is why evaluation cannot be applied only to the final output. Each step in the pipeline needs its own verification. You need test cases for the filing type classifier. You need test cases for the fact extractor. You need test cases for the form population step. And you need integration test cases that verify the pipeline end-to-end because some failure modes only emerge from interactions between steps.
The practical takeaway: decompose your system into its component tasks and evaluate each one independently before evaluating the whole. If you discover that step two has a 15% error rate, that is something you need to address at step two, not something you will notice by staring at final outputs and trying to figure out why some of them feel wrong.
Getting started: using AI to design your evaluation
All of this can sound like a lot of work, and it is. But here is something worth knowing: the same AI tools you are learning to build can help you design the evaluation criteria for those tools. This is not circular. The AI is not evaluating itself. It is helping you, the human designer, think through the dimensions of evaluation that your system requires. You still make every decision. The AI accelerates structured thinking.
I have found this particularly useful in the early stages of a project, when a team knows what they want the system to do but has not yet articulated what correct performance looks like across the full range of inputs. A well-constructed prompt can walk you through the evaluation design process in a single session, producing a draft framework that the team can then review, challenge, and refine. It does not replace the domain expertise needed to determine whether a test case is realistic or whether a threshold is appropriate. It provides the scaffolding so that the domain experts’ time is spent on substance rather than on figuring out what questions to ask.
The following prompt is designed for this purpose. It embodies the principles from this post: evaluation as a front-end design activity, the two-person reliability test, edge case identification, acceptable error calibration, and pipeline decomposition. Copy it, replace the bracketed sections with your specifics, and use the output as a starting point for your evaluation design; not as a finished product.
You are an AI evaluation design consultant helping a legal services team develop evaluation criteria for an AI-assisted tool before development begins.
The tool we are building: [Describe your tool in 2-3 sentences. What does it do? Who uses it? Example: “A retrieval system that helps legal aid attorneys identify relevant court forms and filing instructions based on a client’s described legal situation. Users are staff attorneys and paralegals at a legal aid organization serving self-represented litigants.”]
The specific tasks the tool performs: [List each discrete task. Example: “1. Accepts a natural language description of a client’s legal situation. 2. Identifies the relevant court form(s). 3. Returns the form(s) with filing instructions and deadlines.”]
For each task listed above, work through the following evaluation design steps:
Step 1: The two-person reliability test. For this task, could two trained professionals (e.g., two experienced paralegals, two legal aid attorneys) independently perform it and arrive at the same answer? Identify which tasks would produce high agreement and which would produce frequent disagreement. For tasks with likely disagreement, explain what makes the judgment ambiguous and suggest how the task could be scoped more narrowly or defined more precisely to improve agreement.
Step 2: Test case design at three levels. For each task, draft example test cases at three levels: (a) clear cases where the correct answer is unambiguous, (b) boundary cases where the correct response is genuinely uncertain or where the input partially matches multiple categories, and (c) out-of-scope cases that the system should decline or redirect. For the boundary cases, explain specifically what makes them boundary cases and what decision the team needs to make about how the system should handle them.
Step 3: Acceptable error analysis. For each task, assess the consequence of error. What happens if the system gets this task wrong? Distinguish between tasks where errors have severe, potentially irreversible consequences (e.g., wrong filing deadlines, incorrect form selection that leads to a rejected filing) and tasks where errors are recoverable (e.g., an imperfect initial summary that an attorney will review). Recommend an appropriate performance threshold for each task, grounded in the severity of consequences, and explain your reasoning.
Step 4: Pipeline dependencies. If the system performs multiple tasks in sequence, identify where an error in an earlier task would corrupt downstream tasks. Specify which tasks need independent evaluation checkpoints and which failure modes would only be visible through end-to-end testing.
Step 5: Evaluation method. For each task, recommend how the evaluation should be conducted. Distinguish between tasks that can be evaluated with an answer key (correct/incorrect) and tasks that require human judgment with a rubric. For rubric-based evaluation, flag the risk of evaluator disagreement and recommend how to measure and manage it.
Present the output as a structured evaluation design document that the team can review and refine. Flag any areas where you need additional information from the team to complete the framework.
The output of this prompt is a draft, not a deliverable. It will surface evaluation dimensions you had not considered, propose test cases that expose ambiguity in your specification, and identify pipeline risks you may have overlooked. It will also make incorrect assumptions, propose thresholds that are too high or too low for your context, and generate test cases that do not reflect how your users actually behave. That is fine. The value is in the structured starting point, not in uncritical adoption. Review it with your team, revise it against your domain knowledge, and use it to produce the evaluation framework that the team owns and stands behind.
The technology changes. The models improve. The principle does not: you cannot evaluate what you never defined, and you cannot trust what you never evaluated.