Building an Evaluation Set Before You Trust a Model
A demo tells you the model can succeed once. An evaluation set tells you how often it fails and on what. Build the second one from real inputs before anything you cannot supervise reaches a customer.
On This Page
The demo went well. Somebody typed three questions, the model answered all three, and the room agreed it was ready. That is the moment to stop, because what everyone just observed is that the system can succeed. Nobody learned anything about how often it fails, on what kind of input, or in which direction.
AI evaluation is the discipline of finding that out before a customer does. It does not need a research team or a platform. It needs a few dozen real inputs, a written definition of what a correct output looks like, and the discipline to run them again every time something changes. That is the whole thing. Most teams skip it because it is boring rather than because it is hard.
The position defended here: nothing producing output that a person will act on should reach production without an evaluation set. Not a chat assistant, not a classifier sorting enquiries, not a script drafting replies. The set does not have to be large. It has to be real, and it has to exist before the launch date rather than after the first complaint.
What a demo proves, and what it does not
A demo is a sample of three, chosen by the person who wants it to work. That is not a comment on anyone’s honesty; it is how demos get built. You reach for inputs you understand, phrased the way you would phrase them, about topics you already know are covered.
Real inputs are not like that. They are misspelled, truncated, pasted out of an email with three signatures attached, ambiguous between two products, written in a second language, or about something you stopped selling last year. The distance between demo inputs and real inputs is where every unpleasant surprise lives, and it is not a small distance.
"We tested it for a week and it seemed fine" is not evidence either. Seeming fine is a summary of the cases somebody happened to notice. A held-out set of labelled examples is evidence, because it is the same set every time and two runs can be compared. The standard rises again when the output is published rather than private, because a wrong sentence on your own site can be summarised and repeated onward by search and assistant surfaces long after you fix the page. That is the argument in should AI write your product documentation, and it applies to anything customer-facing.
What an evaluation set actually is
An evaluation set is a list of inputs paired with what the output should be, or with what a correct output must contain. It lives as a file, in version control, next to the code. It is not a spreadsheet somebody keeps locally and it is not a folder of screenshots from the week before launch.
Where the inputs come from
From wherever the system will really be used. Support tickets. Site searches that returned nothing. The free-text field on your enquiry form. Chat transcripts. The questions people email because the site did not answer them. For an assistant that answers from your own content, such as Flidu, the evaluation set is simply the questions visitors actually ask, each one paired with the answer your published content genuinely supports. If you cannot assemble that list, you have found your first result, and it is about your measurement rather than your model.
Do not write the inputs yourself. Invented inputs are demo inputs with extra steps, and they inherit the same blind spots. If the system is genuinely new and no history exists, take the closest adjacent source you have: the sales inbox, the questions asked on onboarding calls, the objections that come up in the second meeting.
Size matters less than coverage
Forty examples chosen for coverage beat four hundred chosen at random. Sample deliberately across the axes that could change behaviour: topic, input length, phrasing, language, and how well your source material supports an answer at all. Include the cases you expect to fail. Include several where the correct behaviour is to decline, or to say plainly that it does not know.
The most valuable examples are the ones where two sensible people would disagree about the label. Those are the ones that force the definition of correct to be written down, and the written definition ends up being worth more than the score it produces.
Deciding what correct means, in writing
This is the part teams skip, and it is the part that decides whether the numbers mean anything at all. Before looking at a single output, write down what a passing answer must contain and what disqualifies it. Five lines is enough. A rubric that exists is the point, not a rubric that is elegant.
Write the expected answer for each input before you run the system. If you label afterwards, having read the output, you will label generously and you will not notice yourself doing it. This single ordering decision is what separates an evaluation set from a collection of things that looked about right on the day.
Binary where you can, graded where you must
Prefer binary judgements. Did it include the policy exception, yes or no. Did it produce valid JSON with every required field, yes or no. Binary checks can be automated, they do not drift between reviewers, and they are the only kind you can run on every commit without occupying a person. Keep graded scoring for things that genuinely have degrees, such as whether an answer is complete, and write the levels out so that two reviewers land in the same place.
Where a person has to judge, make the judging a defined job rather than a favour asked on a Friday. Our parent company describes that discipline in the reviewer’s pass, and the same principle carries over: a reviewer with a rubric and a fixed sample is doing quality control, while a reviewer skimming outputs is doing reassurance.
What AI evaluation should measure
One accuracy number hides everything worth knowing. Split failures by class, because the classes have different costs, different fixes, and different people who need to hear about them.
| Failure class | What it looks like | How to score it |
|---|---|---|
| Wrong answer | States something the source does not support, confidently | Binary, against the labelled expectation |
| Missing answer | Refuses or hedges when the source does cover it | Binary, counted separately from wrong |
| Partial answer | Correct but omits a condition that changes the decision | Graded, with a written rubric |
| Format break | Right content, unusable shape: invalid JSON, wrong field | Binary, machine-checked |
| Policy break | Makes a promise, quotes a price, gives advice you do not give | Binary, flagged by a reviewer |
| Latency and cost | Correct, but too slow or too expensive at volume | Measured rather than judged |
Separate the dangerous classes from the annoying ones. A confidently wrong statement about a refund policy is a different event from a clumsy sentence, and rolling them into one figure means the dangerous one gets averaged away by the harmless one. This split is also how you decide where a person has to sit in the path, which is the subject of designing a review step that actually catches errors.
Running it, and what a run costs
The mechanics are undramatic, which is rather the point of them.
- Freeze the set. Commit the inputs and the expected outputs to the repository. Changing the set and the system in the same week means you cannot attribute the difference to either of them.
- Record the configuration. Model, version, prompt, retrieval source, temperature, tool definitions. A result reported without its configuration is a rumour with a percentage attached.
- Run it automatically. On every change to the prompt, the retrieval source or the model, run the machine-checkable half inside the pipeline and treat a regression exactly as you would treat a failing test.
- Sample the human half. Graded cases need a person. Fix the sample size and the cadence, and rotate reviewers so that one individual’s tolerance does not quietly become the company standard.
- Publish the failures, not the score. The useful artefact of a run is the list of inputs that failed and how they failed. A single percentage gives the room nothing it can act on.
Keep the results next to the rest of your monitoring rather than in a document nobody opens. An evaluation run is a health check with a slower heartbeat, and it belongs alongside the signals you already watch, in the spirit of observability for a site that only breaks occasionally.
Keeping the set honest over time
An evaluation set decays in three specific ways. All three are avoidable, and all three are much easier to prevent than to detect after the fact.
First, contamination. Tune the prompt against the whole set and the score stops measuring generalisation; it starts measuring how well you have memorised your own examples. Hold back a slice you never look at during development, and open it only when you believe the work is finished.
Second, drift. Real inputs change. New products, new phrasing, a campaign that brings a different audience through the door. Add examples from the last quarter on a fixed schedule and retire the ones about things you no longer sell. A set that has not changed in a year is measuring last year.
Third, the set becomes the target. Once a number is reported upward, the pressure is to move the number, and the cheapest way to move it is to fit the examples. Guard against that with the held-out slice, by rotating who chooses new examples, and by reporting failure classes instead of a single headline figure that invites exactly this behaviour.
Split the set the day you build it. Two thirds for development, one third held back and untouched. If the two halves disagree at the end, you have been fitting to the examples rather than fixing the system, and it is far better to learn that a week before launch than from a customer three months after it.
Be careful about what a short run can honestly tell you. Improvements measured over a fortnight, on a set assembled from one channel in one month, will overstate themselves and everyone in the room will want to believe them. Our parent company makes the sober version of that argument in what ninety days can honestly show, and the caution transfers directly.
The inputs you would rather not think about
A set built only from cooperative users is half a set. Include inputs that try to make the system do something it should not: requests for information it holds but must not disclose, instructions buried inside pasted text, attempts to extract a commitment on your behalf. Anything that accepts untrusted input and then acts on it inherits the ordinary catalogue of web application risks, and the model layer adds a fresh way to deliver them.
Decide what a correct refusal looks like, then test that refusals actually happen. A system that produces an answer to everything is not being helpful, it is being unguarded. That line matters most where an assistant sits in the place a form used to occupy, which is the question behind can an AI assistant replace your contact form.
Publish the boundary as well as testing it. Our own position on where automated systems may and may not act is written down on our responsible AI page, and the discipline is worth copying whatever your policy says: an unwritten boundary is a boundary that moves under pressure, usually on the day it matters.
What this costs, and where it is not worth it
Assembling a first set takes a day or two of somebody who knows the domain, and the labelling is the slow half. Running it costs whatever the model costs per call, multiplied by the number of examples, multiplied by how often you run it. People forget that third multiplier, and it is a good reason to keep the set small and the automated half cheap enough to run without a conversation.
The honest concession: for a genuinely low-stakes internal task, where a wrong output is noticed immediately by the person who asked for it and costs nothing to redo, a formal set may not be worth building. Summarising your own meeting notes does not need a rubric. Below that line, get on with the work.
A set earns its cost once any one of these is true:
- A wrong output can reach somebody outside the team who will believe it.
- A wrong output triggers something awkward or expensive to undo.
- More than one person changes the prompt, the model or the retrieval source.
- You have already had the argument about whether a change made things better.
There is also a standing cost, which is that a set nobody owns stops being run within about two releases and nobody announces it. That is the same failure described in the standing cost of an automation nobody owns, and it has the same remedy: a name against it, and a place in a pipeline rather than a place in somebody’s calendar.
A model that is right most of the time is a feature. A model that is wrong in ways you can predict and bound is a system. The evaluation set is what gets you from the first to the second.
The rule we apply
Nothing ships without a set. Not as a policy to be proud of, but because the alternative is learning the failure distribution through customer complaints, which is slower, more expensive and considerably more embarrassing. If you are starting an AI development or integration project this month, the first artefact is not a prompt. It is forty labelled examples and a page of rubric.
The order we would run it: collect real inputs, write the rubric, label without looking, run once and read every single failure, then decide whether the system is worth building at all. That last step is real and it is not decoration. Some evaluations end the project, and that is the cheapest outcome available to anyone. The same sequence applies to any automation or integration that acts without a person watching it act.
And the condition that reverses the advice: if the output is disposable, private, and checked immediately by the person who asked for it, skip the ceremony. Everything else gets a set. If you want a second pair of eyes on what to measure for a particular use case, tell us what you are automating and what a wrong answer would cost you.
Common questions.
How many examples does an evaluation set need?
Fewer than most people expect, if they are chosen well. Forty to a hundred examples selected for coverage will surface most systematic failures, where several hundred random samples mainly repeat the easy cases. Sample deliberately across topic, phrasing, input length, language, and how well your source material supports an answer, and include cases where the correct behaviour is to decline or to say it does not know.
What should I measure when testing an AI feature?
Measure failure classes separately rather than one accuracy score. Wrong answers, missing answers the source did cover, partially correct answers that omit a condition, broken output formats, and policy breaks such as making a promise or quoting a price all carry different costs and different fixes. Track latency and cost per call alongside them, because a correct answer that is too slow or too expensive is still unusable.
Can I use another model to grade the outputs?
Yes for the cheap repetitive checks, and with caution elsewhere. Automated grading works well for format validation, the presence of a required fact, and obvious contradictions of the source material. It is unreliable for judgements about tone, completeness and risk, and it tends to share blind spots with the system being tested. Validate the grader against human labels on a sample first, then re-check it periodically.
How often should an evaluation set be run?
Run the automated portion on every change to the prompt, the retrieval source, the model or its version, inside the same pipeline as your tests. Run the human-judged portion on a fixed cadence, with monthly a reasonable default and a fixed sample size so results stay comparable. Refresh the examples themselves quarterly by adding recent real inputs and retiring ones about products you no longer offer.
What is a held-out set and why does it matter?
A held-out set is the portion of your examples you never look at while tuning. Its purpose is to catch the case where apparent improvements are really adjustments fitted to the examples you have been staring at for a fortnight. Split roughly two thirds for development and one third held back, then open the held-back portion only when you believe the work is done. Disagreement between the two is the signal.
Should the set include attempts to misuse the system?
Yes, and from the first version rather than as a later addition. Include requests for information the system holds but should not disclose, instructions hidden inside pasted text, and attempts to make it commit to something on your behalf. Decide what a correct refusal looks like and test that refusals happen in practice. A system that produces an answer to absolutely everything is unguarded rather than helpful.
When is a formal evaluation set not worth building?
When the output is disposable, private, and checked immediately by the person who asked for it. Summarising your own notes does not need a rubric. The threshold is whether a wrong output can reach somebody who will believe it, or can trigger an action that is awkward to reverse. Once either is true, building a small set costs far less than discovering the failure pattern from complaints.
Facing this in your
own business?
Tell us where you’re headed — we’ll map the shortest honest route.