This is a build-along. By the end you will have two files on disk, twenty labelled cases, one scorer written in code, a second scorer that is a model, and a number you can paste into a commit message. Every file you need is printed in full below, so you can follow along by copying blocks in the order they appear.
An eval is a repeatable, scored test of your system on your own data. What Are AI Evals? is the definition, and AI Evals 101 is the framework behind most of the choices made here. This post does not re-explain either. It only does the work.
What you are building
The system under test takes a teacher's free-text request and turns it into a quiz spec: a grade, a subject, a number of questions, and a topic. A request like "10 questions on photosynthesis, 7th grade science" should come back as a small JSON object that the rest of a product can act on.
That task was chosen because its output splits cleanly in two, and most real outputs do. Three of the four fields have exactly one correct value, so code can check them. The fourth, the topic, has no single right phrasing, so code cannot. Building evals gets a lot easier once you sort your output into those two piles first, because each pile has an obvious scorer and mixing them gives you neither.
The stack is Python 3.10 or newer with the official Anthropic SDK, and nothing else. No eval framework, no test runner, no service. A first eval should be a script you can read top to bottom in one sitting, and this one is about 130 lines.
Setting up
python3 -m venv .venv
source .venv/bin/activate
pip install anthropic
export ANTHROPIC_API_KEY=your-key-hereTwo files live side by side: first_eval.py and cases.jsonl. Every Python block below goes into first_eval.py, in the order it appears, with one clearly marked exception. Nothing is elided and nothing depends on scaffolding you cannot see.
One warning before you run anything: an eval costs real money. The default split here is 13 cases and each case makes up to two API calls, so a run is 26 requests. Requests are not the bill, though: you pay for thinking tokens too, so adding "effort": "low" to the existing output_config dict alongside "format", as in output_config={"format": {...}, "effort": "low"}, is the first lever if a run costs more than the task warrants. None of that matters at 26 requests. It matters at 500 cases on every commit, and it is worth knowing the shape of that bill before you get there.
The thing you are testing
"""A complete eval for one model call. Run: python first_eval.py dev"""
import json
import sys
from pathlib import Path
import anthropic
client = anthropic.Anthropic()
MODEL = "claude-opus-5"
JUDGE_MODEL = "claude-opus-5"
SYSTEM = """You turn a teacher's request into a quiz spec.
Reply with one JSON object and nothing else, using exactly these keys:
grade, subject, question_count, topic.
grade is a US grade number from 1 to 12.
subject is one of math, science, english, history.
question_count is a whole number.
topic is a short phrase naming what the questions are about.
Use null for any field the teacher did not state or clearly imply."""
def text_of(message):
"""Join the text blocks of a response. Thinking blocks are skipped."""
return "".join(block.text for block in message.content if block.type == "text")
def run_target(request):
"""Return (spec, None) on success, or (None, raw_text) if the reply is not an object."""
message = client.messages.create(
model=MODEL,
max_tokens=16000,
system=SYSTEM,
messages=[{"role": "user", "content": request}],
)
raw = text_of(message)
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
return None, raw
return (parsed, None) if isinstance(parsed, dict) else (None, raw)Three details in there are load-bearing.
text_of walks the content blocks instead of reaching for message.content[0].text. A response is a list of typed blocks, and on a thinking-capable model the first one is often not the text. Indexing works right up until it does not.
run_target returns the raw string instead of raising, for both ways a reply can fail to be a spec: text that is not JSON, and JSON that is not an object. A bare quoted refusal is valid JSON, so without the isinstance check your first out-of-scope case takes the run down with an attribute error. Either way, a reply your eval cannot read is a result to count and show you, not a crash that ends the run on case three.
And the prompt is deliberately the plain version, the one you would actually have written before you had an eval. It is also not pinned to a schema the way the judge below is, because that would drive parse failures to zero by construction and delete a signal you are about to be told to watch. Improving it is the last section. There is no point tuning a prompt while you still have no way to tell whether the tuning helped.
Building the eval set
The eval set is the part people skip, and it is the part that decides whether any of the rest is worth running. A scorer measures against the cases you gave it, so an unrepresentative eval set produces a confident, precise, useless number.
How many cases to start with
Twenty. Below about ten, a single case swings the score by ten points and you cannot separate a real regression from the model sampling differently, so you end up chasing noise. Above about fifty, you will spend a week labelling and never actually run the thing. Twenty cases is enough to embarrass you and small enough to hand-label in an afternoon.
Twenty is a starting size, not a target. Eval sets that work are the ones that grow every time production surprises you.
Where the cases come from
In rough order of value:
- Production logs. Real inputs from real users beat anything you can invent, because your users phrase things in ways you would not have thought to test.
- Your bug tracker and support threads. Every complaint about the feature is a case, already written up, already known to have failed once.
- Hand-written cases for behaviour you have not shipped yet, or cannot wait to observe.
- Deliberate edge cases: the empty input, the out-of-scope request, the ambiguous one, the one where a field is simply absent.
Aim for roughly half ordinary traffic, a quarter known-hard cases, and a quarter deliberate edges. If everything passes on the first run, you built a set that is too easy to teach you anything, and the fix is to go and find harder inputs rather than to celebrate.
Label the cases by hand. Never ask the system under test what the right answer is: a set labelled by the thing you are testing cannot catch that thing being wrong.
The file
Here is the whole set. Save it as cases.jsonl, one JSON object per line.
{"id": "t01", "split": "dev", "request": "10 questions on photosynthesis, 7th grade science", "grade": 7, "subject": "science", "question_count": 10}
{"id": "t02", "split": "dev", "request": "I need a quick 5 question warm up on fractions for my 5th graders", "grade": 5, "subject": "math", "question_count": 5}
{"id": "t03", "split": "dev", "request": "Can you make something on the water cycle for grade 4?", "grade": 4, "subject": "science", "question_count": null}
{"id": "t04", "split": "dev", "request": "15 questions about the American Revolution for my sophomores", "grade": 10, "subject": "history", "question_count": 15}
{"id": "t05", "split": "dev", "request": "8 questions on subject verb agreement, 6th grade ELA", "grade": 6, "subject": "english", "question_count": 8}
{"id": "t06", "split": "dev", "request": "a dozen questions on the periodic table for 9th grade chemistry", "grade": 9, "subject": "science", "question_count": 12}
{"id": "t07", "split": "dev", "request": "quiz on Romeo and Juliet act 2 for 10th grade english, 12 questions", "grade": 10, "subject": "english", "question_count": 12}
{"id": "t08", "split": "dev", "request": "20 questions on long division for elementary students", "grade": null, "subject": "math", "question_count": 20}
{"id": "t09", "split": "dev", "request": "My middle schoolers keep mixing up mean and median. Something short would help.", "grade": null, "subject": "math", "question_count": null}
{"id": "t10", "split": "dev", "request": "6 questions, 8th grade, linear equations", "grade": 8, "subject": "math", "question_count": 6}
{"id": "t11", "split": "dev", "request": "Practice on the causes of World War One for my 11th graders, about 10 questions", "grade": 11, "subject": "history", "question_count": 10}
{"id": "t12", "split": "dev", "request": "Hi, can you print worksheets for me?", "grade": null, "subject": null, "question_count": null}
{"id": "t13", "split": "dev", "request": "We are on cell division in the biology textbook and I have 25 minutes left with my 9th graders on Friday. Ten questions would be perfect.", "grade": 9, "subject": "science", "question_count": 10}
{"id": "t14", "split": "holdout", "request": "12 questions on the Civil Rights Movement for 8th grade social studies", "grade": 8, "subject": "history", "question_count": 12}
{"id": "t15", "split": "holdout", "request": "Something on adjectives for 3rd grade", "grade": 3, "subject": "english", "question_count": null}
{"id": "t16", "split": "holdout", "request": "9 questions about states of matter for grade 6", "grade": 6, "subject": "science", "question_count": 9}
{"id": "t17", "split": "holdout", "request": "Need a 10 question review of the unit circle for precalc juniors", "grade": 11, "subject": "math", "question_count": 10}
{"id": "t18", "split": "holdout", "request": "5 quick ones on capital letters and full stops, second grade", "grade": 2, "subject": "english", "question_count": 5}
{"id": "t19", "split": "holdout", "request": "My 12th graders are reading Beloved. 15 questions on chapters one through four.", "grade": 12, "subject": "english", "question_count": 15}
{"id": "t20", "split": "holdout", "request": "Can you do 30 questions on everything we covered this term?", "grade": null, "subject": null, "question_count": 30}Notice what is and is not in there. Each case carries expected values for grade, subject, and question_count, and no expected value for topic, because there is no single correct phrasing of a topic and pretending otherwise would just build a scorer that fails good answers.
Several cases expect null, and those are the ones that earn their place. Case t03 names no number of questions, so inventing a plausible ten is a failure and not a nicety. Case t12 is not a quiz request at all. Case t20 asks for thirty questions on nothing in particular. Meanwhile t04, t17, and t18 write the grade in words rather than digits, which is the kind of thing a model handles well until the day it does not.
Cases go in a file, not in the test code, and the file goes in version control. You will want to diff it later, and a reviewer should be able to see a case being added without reading a diff of Python.
Loading it is four lines:
CASES = Path(__file__).with_name("cases.jsonl")
def load_cases(split):
lines = CASES.read_text(encoding="utf-8").splitlines()
rows = [json.loads(line) for line in lines if line.strip()]
return [row for row in rows if row["split"] == split]Splitting the eval set, and why you hold cases back
The split field divides the twenty cases into 13 development cases and 7 held out. The development split is the one you run constantly, stare at, and argue with. The holdout is the one you run when you are about to ship.
The reason for the divide is that you are going to tune against whatever you can see. Not because you are cheating, but because that is what iterating means: you read the failing case, you add a clause to the prompt, the case passes. Do that fifteen times and you have a prompt fitted to fifteen specific inputs and a dev score that no longer predicts anything. The holdout is the only number that stays honest, and it only stays honest because you look at it rarely.
At twenty cases, split by hand rather than at random, and check the composition. Each split here carries at least one case with a missing question count, one where the grade is implied rather than stated, and one that is not really a request for a quiz. A holdout that accidentally collected only the easy cases will happily tell you everything is fine.
Two rules keep the split useful. Do not read holdout failures while you are iterating: run it, record the number, close the file. And when a holdout case does expose a failure mode you want to fix, move that case into dev before you fix it, then write a fresh holdout case covering the same behaviour. A holdout case you have optimised against is a dev case wearing a costume.
Hand-splitting stops scaling somewhere in the low hundreds. When it does, replace the split field with a hash of the case id, so that the assignment is stable and appending new cases never reshuffles the old ones:
import hashlib
def split_for(case_id):
digest = hashlib.sha256(case_id.encode("utf-8")).hexdigest()
return "holdout" if int(digest[:8], 16) % 10 < 3 else "dev"That block is the marked exception: it is not part of first_eval.py. It is what you reach for on the day hand-curation becomes the bottleneck.
Writing the scorers
The code scorer handles the three fields with exactly one right answer:
FIELDS = ("grade", "subject", "question_count")
def normalise(value):
"""Fold away the differences you do not care about, and only those."""
return value.strip().lower() if isinstance(value, str) else value
def check_fields(case, spec):
return {key: normalise(spec.get(key)) == normalise(case[key]) for key in FIELDS}The comparison is strict on purpose. If the model returns the string "5th" where the schema says the integer 5, that is a failure, because the code downstream of this call will break on it. Softening the comparison to make that pass would be measuring your own patience rather than the system.
The normalising is equally deliberate, and narrow. It folds case and surrounding whitespace, so "Math" and "math" are the same answer, because they genuinely are. It folds nothing else. An evaluator that fails a correct output is a bug in the evaluator, and it costs you more than a missing test does: you will spend a morning fixing a model that was right.
The topic needs a judge, because there is no reference string to compare against:
JUDGE_SCHEMA = {
"type": "object",
"properties": {
"verdict": {"type": "string", "enum": ["pass", "fail"]},
"reason": {"type": "string"},
},
"required": ["verdict", "reason"],
"additionalProperties": False,
}
JUDGE_PROMPT = """A teacher wrote this request:
{request}
A system extracted this topic from it: {topic}
Answer pass if the topic is faithful to the request and specific enough to
write questions from. Answer fail if it is broader than the request, if it
adds material the teacher did not ask for, or if it is missing while the
request names one. If the request names no topic at all, null is correct.
Give one short sentence of reasoning."""
def judge_topic(request, topic):
message = client.messages.create(
model=JUDGE_MODEL,
max_tokens=16000,
messages=[
{
"role": "user",
"content": JUDGE_PROMPT.format(
request=request, topic=json.dumps(topic)
),
}
],
output_config={"format": {"type": "json_schema", "schema": JUDGE_SCHEMA}},
)
return json.loads(text_of(message))The schema in output_config is what stops the judge from becoming the flakiest part of your eval. Without it you are regex-hunting for the word "pass" in a paragraph, and a run fails because the judge got chatty. With it, the response is valid JSON in the shape you asked for, every time.
The judge prompt is short, and it sees only the request and the extracted topic. Do not hand a judge the whole trace and hope. One criterion, the minimum context needed to decide it, and a sentence of reasoning you can read when you disagree with the verdict.
Note that the judge is asked for a verdict on one narrow question, not for a quality score out of ten. Numeric ratings from a model look precise and drift constantly. A binary verdict against a written rule is reproducible enough to compare across runs.
JUDGE_MODEL and MODEL start out as the same model, which means the model grades its own output, and a mild self-preference bias makes the topic pass rate a little generous. That is an acceptable trade for a first eval, and the constant is kept separate so you can point the judge elsewhere when it starts to matter.
Wiring it together
def main():
split = sys.argv[1] if len(sys.argv) > 1 else "dev"
cases = load_cases(split)
if not cases:
sys.exit("no cases in split " + repr(split))
results = []
field_hits = 0
topic_hits = 0
parse_failures = 0
for case in cases:
spec, raw = run_target(case["request"])
if spec is None:
parse_failures += 1
results.append({"id": case["id"], "parsed": False, "raw": raw})
print(case["id"] + " PARSE FAILURE")
continue
fields = check_fields(case, spec)
field_hits += sum(fields.values())
verdict = judge_topic(case["request"], spec.get("topic"))
topic_ok = verdict["verdict"] == "pass"
topic_hits += int(topic_ok)
results.append(
{
"id": case["id"],
"parsed": True,
"spec": spec,
"fields": fields,
"topic_verdict": verdict,
}
)
missed = [key for key, ok in fields.items() if not ok]
line = "{} fields {}/{} topic {}".format(
case["id"], sum(fields.values()), len(FIELDS), verdict["verdict"]
)
print(line + (" missed " + ", ".join(missed) if missed else ""))
scored = len(cases) - parse_failures
print("")
print("split " + split)
print("cases {}".format(len(cases)))
print("field accuracy {}/{}".format(field_hits, scored * len(FIELDS)))
print("topic pass rate {}/{}".format(topic_hits, scored))
print("parse failures {}".format(parse_failures))
out = Path(__file__).with_name("results-" + split + ".json")
out.write_text(json.dumps(results, indent=2), encoding="utf-8")
print("wrote " + out.name)
if __name__ == "__main__":
main()That is the whole script. Two things about it are choices rather than plumbing.
It reports three numbers instead of one. A single blended pass rate feels tidy and tells you nothing on the day it drops, because field extraction and topic quality fail for unrelated reasons and a combined score hides which one moved. Keep criteria separate for as long as you can stand it.
And it writes every per-case result to disk, including the judge's reasoning and the raw text behind a parse failure. The printed summary is for you today. The file is for the version of you who, three weeks from now, wants to know exactly what case t09 used to return.
Running it
python first_eval.py devThe output takes this shape, with illustrative figures rather than measured ones:
t01 fields 3/3 topic pass
t02 fields 2/3 topic pass missed grade
t03 fields 3/3 topic fail
...
split dev
cases 13
field accuracy 35/39
topic pass rate 10/13
parse failures 0
wrote results-dev.jsonTwo runs of your own will not always agree with each other either, because the model is sampling. That instability is the entire argument for writing an eval instead of trying three inputs by hand and forming an impression. Record the summary along with the model id and the date, in the commit message or a plain text log, and you have a baseline.
Reading the first result
Resist reading only the score. Open results-dev.json and sort the failures into three piles. They are three cells of the output-against-evaluator grid that AI Evals 101 sets out in full; here they are just triage.
- The system is wrong and the eval says wrong. These are real defects and the reason you did this.
- The system is right and the eval says wrong. That is a defect in your criterion rather than in the model, and you fix it first, because every future run inherits the error.
- The system is wrong and the eval says right. The worst pile, and the hardest to find, because nothing is on fire. You usually discover these by reading passing cases at random.
Parse failures deserve their own look. A case whose reply your eval could not read is never scored at all, so a rising parse-failure count next to a stable field accuracy is a real regression hiding behind a flat-looking number.
Change one thing and re-run
Now the loop that makes an eval worth having. Change exactly one thing, re-run dev, and compare.
A reasonable first change here is to close the gap the null cases exposed. Add two lines to SYSTEM:
If the teacher does not state a number of questions, question_count must be null.
Do not fill a field with a plausible default.Re-run python first_eval.py dev, then diff the new results-dev.json against the old one. Because the file is keyed by case id and written in a stable order, the diff tells you precisely which cases changed verdict, which is the interesting question. The summary alone cannot: two cases fixed and two cases broken shows up as no change at all.
Change one thing at a time. If you edit the prompt, add a retry, and swap the model in one go, you get one number and no idea which of the three moved it.
When dev looks good, run the holdout once:
python first_eval.py holdoutIf dev improved and holdout did not, you tuned the prompt to the dev cases rather than to the task. That is not a disaster and it is not a moral failing. It is information you could not have got any other way, and it is precisely what the seven held-back cases were for.
Where to go next
You have a working eval. Grow it in this order.
- Add cases before you add scorers. A larger eval set improves every number you already collect, while a new criterion only measures something new.
- Feed it from production. Every bug report about this feature becomes a case, with the corrected answer as its label. That is the loop that turns a static file into a real regression suite.
- Calibrate the judge before you trust it. Label twenty topics by hand, run the judge on the same twenty, and look at where you disagree. Fix the judge prompt until it agrees with you, then let it run unattended.
- Run the dev split in CI on any change to the prompt, and let the holdout stay a manual pre-release step.
- Parallelise when 26 sequential calls starts to annoy you. A thread pool over the case list is a ten-line change, and it is worth doing only once the wait is what stops you from running it.
The broader picture behind all of that, including offline versus online evaluation, curating production traces, and where human review is non-negotiable, is in AI Evals 101.
What you have now is the thing most teams discuss and never build: a number, on your own data, that moves when you change something. Twenty cases and one afternoon is all it costs, and every subsequent decision about this feature is cheaper because of it.