Process
Factory evals
You might say: eval suite, regression evals, replayed tickets
A fixed set of past tickets and scenarios that the factory reruns after any change to its models, prompts, tools or gates. Regression tasks should keep passing at close to 100%. Capability tasks show whether the change made the factory better at work it used to fail. Each eval names a failure mode and counts how often it happens.
- What it does
- Shows whether a change to the factory helped or hurt before the change meets real tickets.
- Ask
- Which past tasks prove the factory still works, and what result blocks a change to the factory?
What to look for
- A suite of past tickets with known good outcomes
- Pass rates per model, prompt or gate version
- A factory change held back by a failing regression eval
Connections
- See also: Improvement loop
- See also: Model choice
Sources
- Anthropic, Demystifying evals for AI agents, Demystifying evals for AI agents
Once evals exist, you get baselines and regression tests for free
- Anthropic, Demystifying evals for AI agents, Demystifying evals for AI agents
Regression evals ask, ‘Does the agent still handle all the tasks it used to?’ and should have a nearly 100% pass rate.
- Chip Huyen, AI Engineering: Building Applications with Foundation Models, Agents, adapted from the Agents section of the book (author's blog)
To evaluate an agent, identify its failure modes and measure how often each of these failure modes happens
Use it on your project
teardown-apply: hold "Factory evals" (factory-evals) from the teardown at https://teardownatlas.com/software-factory/ against my project
Paste it into a coding agent that has the teardown skills.
An AI agent wrote this map with the teardown skill. It can contain errors: check the sources before you rely on a part. About and legal