Concept

Non-determinism

You might say: nondeterminism, run-to-run variance, different answer every run

The same ticket on the same code can produce a different change on each run.

What it does: Makes one passing run weak evidence, so the factory repeats scenarios and reports scores across several trials.
Ask: How many trials passed, and did each run reach the same user outcome?
Typical: Use pass@k when one successful attempt is enough, and pass^k when every trial must succeed.
Trouble: A change passes one trial and fails or behaves differently on another, or a score is reported from one trial only.

Sources

Matt Pocock, AI Coding Dictionary, Non-determinismThe same input can produce different output. Run a model twice with identical context and you may get two different answers — sometimes a word, sometimes a completely different approach. Nothing in your code has to change for this to happen.
Anthropic, Demystifying evals for AI agents, The structure of an evaluationEach attempt at a task is a trial. Because model outputs vary between runs, we run multiple trials to produce more consistent results.
Anthropic, Demystifying evals for AI agents, How to evaluate AI agentspass@k measures the likelihood that an agent gets at least one correct solution in k attempts.
Anthropic, Demystifying evals for AI agents, How to evaluate AI agentspass^k measures the probability that all k trials succeed.
Justin McCarthy (StrongDM), Software Factories And The Agentic Moment, From Tests to Scenarios and SatisfactionWe use the term satisfaction to quantify this validation: of all the observed trajectories through all the scenarios, what fraction of them likely satisfy the user?

teardown-apply: hold "Non-determinism" (non-determinism) from the teardown at https://teardownatlas.com/software-factory/ against my project

Paste it into a coding agent that has the teardown skills.

An AI agent wrote this map with the teardown skill. It can contain errors: check the sources before you rely on a part. About and legal