Part

Holdout scenario

You might say: hidden test, holdout set

A scenario kept where the coding agent cannot read it. Only the verifier runs it, so the author cannot shape its code to pass that test alone.

What it does
Catches an agent that wrote code to pass the tests it could see, and not to do the job.
Ask
Which scenarios can the author run read, and which ones does only the verifier hold?

What to look for

  1. Scenarios stored outside the repository the agent works in
  2. A verifier run that reports results the author never saw
  3. The loop ends only when the hidden scenarios pass

Pattern

Holdout set

Sources

  1. Simon Willison, How StrongDM’s AI team build serious software without even looking at the codeThat idea of treating scenarios as holdout sets … used to evaluate the software but not stored where the coding agents can see them … is fascinating.
  2. Justin McCarthy (StrongDM), Software Factories And The Agentic Moment, The PrinciplesThe loop runs until the holdout scenarios pass (and stay passing)
  3. Justin McCarthy (StrongDM), Software Factories And The Agentic Moment, Software Factories And The Agentic MomentWe repurposed the word scenario to represent an end-to-end "user story", often stored outside the codebase (similar to a "holdout" set in model training)

Use it on your project

teardown-apply: hold "Holdout scenario" (holdout-scenario) from the teardown at https://teardownatlas.com/software-factory/ against my project

Paste it into a coding agent that has the teardown skills.

An AI agent wrote this map with the teardown skill. It can contain errors: check the sources before you rely on a part. About and legal