Pattern
Holdout set
You might say: test set, held-out data, hidden test
Keep part of the evidence away from whatever is being tuned, and judge the result only on that hidden part. A score on examples the learner has already seen says little about how it will do on new ones.
- Ask
- What evidence has the learner never seen, and who holds it?
Sources
- scikit-learn 1.5, Cross-validation: evaluating estimator performance, 3.1. Cross-validation: evaluating estimator performance
Learning the parameters of a prediction function and testing it on the same data is a methodological mistake
- scikit-learn 1.5, Cross-validation: evaluating estimator performance, 3.1. Cross-validation: evaluating estimator performance
it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set
- Simon Willison, How StrongDM’s AI team build serious software without even looking at the code
That idea of treating scenarios as holdout sets … used to evaluate the software but not stored where the coding agents can see them … is fascinating.
Use it on your project
teardown-apply: hold "Holdout set" (pattern-holdout-set) from the teardown at https://teardownatlas.com/software-factory/ against my project
Paste it into a coding agent that has the teardown skills.
An AI agent wrote this map with the teardown skill. It can contain errors: check the sources before you rely on a part. About and legal