10 Designing Experiments That Can Be Checked
10.1 Learning objectives
After completing this unit, you will be able to:
- turn a broad question into a testable hypothesis and outcome measure;
- distinguish the variable being changed, the outcome being measured, and the conditions being controlled;
- set the parameter grid, number of replications, and stopping rule before seeing the results;
- explain what a pseudorandom seed can and cannot reproduce;
- separate development data from held-out test data (holdout);
- report effect sizes and negative results without changing the target afterward; and
- limit experimental conclusions so that they are not treated as general proofs.
Local prerequisites: Units 1–4, percentages, means, comparisons of fractions, and simple step functions at A30 level. No calculus or inferential statistics is required.
10.2 Questions first, results second
The Unit 7 experiment studies a simple decision rule. Each case has an integer score from 0 to 9. For a threshold , the program predicts a positive outcome if
We choose one threshold from the grid . The question fixed before running the experiment is:
Does the threshold selected on development data improve accuracy on holdout data by at least 0.10 compared with the reference threshold of 6?
The value 0.10 means ten percentage points, not “ten percent of the previous accuracy.” The initial hypothesis answers “yes.” The experiment may then support or fail to support that answer; a hypothesis is not a promise about the result.
This question states three things that are often missing:
- the comparison rule: a threshold of 6;
- the outcome measure: the proportion of correct predictions; and
- the minimum effect considered meaningful: 0.10.
Without all three, it is easy to choose a story after seeing the numbers.
10.3 Variables, controls, and confounders
In this experiment:
| Role | Content |
|---|---|
| variable being changed | decision threshold |
| outcome being measured | accuracy: the number of correct predictions divided by the number of cases |
| controlled conditions | case generator, sample size, replications, and seed schedule |
| random factors | the synthetic score and outcome for each case |
A confounder is another factor that changes along with the main variable, making the cause of a difference unclear. If threshold 3 is tested on easy cases while threshold 7 is tested on difficult cases, the difference in accuracy may come from the cases rather than the threshold.
The script prevents this confounding through a paired design: within a replication, every threshold is evaluated on exactly the same cases. Only the threshold changes. This design does not remove every possible error; the case-generation rule itself may still be poor. It does, however, isolate the comparison being made.
10.4 Parameter grid and selection rule
The parameter grid is fixed as
The program does not add new thresholds after seeing the results. Every candidate completes 20 development replications, each containing 200 cases. The threshold with the largest total number of correct predictions is chosen. If two thresholds tie, the smaller threshold wins. This tie-breaking rule may seem trivial, but without a fixed rule, two people running the experiment may select different results.
Grid search compares only these five candidates. If threshold 5 wins, a valid conclusion is “5 is best on this grid and these development data according to the specified measure,” not “5 is the best possible threshold for every population.”
A very dense grid is not a free benefit, either. The more choices tried on the same data, the greater the chance of finding one that fits peculiarities of the development data by chance. This is one reason for using a holdout.
10.5 Seeds and nondeterminism
random.Random(seed) generates a pseudorandom sequence. The same seed in the same environment produces the same sequence, allowing a failure to be rerun. This unit uses base seed 1729 for development and 271828 for the holdout. Replication uses base_seed + i.
A seed does not turn an experiment into a proof, and it cannot capture every source of real-world nondeterminism. Examples include:
- physical sensor noise;
- changes in temperature or voltage;
- the order of parallel processes;
- data packets arriving over a network; and
- human actions during measurement.
For these sources, record the time, equipment, calibration, treatment order, environment, and raw data as relevant. Independent replication is still needed. Writing seed=42 does not reproduce a laboratory.
10.6 Replication and stopping rules
A single run may favor one threshold by chance. Replication shows whether the result persists across several samples. The unit’s plan specifies:
- development: exactly cases; and
- holdout: exactly cases.
The word “exactly” is part of the method. The program does not stop when the results first look good. If we keep inspecting results and stop at a pleasing replication, the data-collection rule depends on the outcome; the final summary is then more optimistic than the original plan warrants.
A stopping rule may specify a fixed number of cases, a fixed time limit, or another criterion. What matters is that it is written before results are inspected and reported together with any deviations.
10.7 Development data and holdout data
Development data are used to compare the grid and select a threshold. Holdout data are generated with a separate seed schedule, do not influence the selection, and are inspected once after the threshold has been fixed.
A simple analogy is practice and an examination. If examination answers are used to change how you answer, and the score on that same examination is then reported as the final assessment, the examination has become practice. Likewise, if holdout results are inspected repeatedly to change the grid, features, or rules, the holdout is no longer an untouched assessment. We need a new holdout, or we must honestly describe that stage as further development.
This separation does not make the results applicable to every real-world setting. The holdout data still come from a particular case generator. The separation merely reduces one kind of overfitting to the development data.
10.8 Effect size, not visual impression
In the reference record, threshold 5 is selected. On the holdout, threshold 5 is correct on 2,355 of 3,000 cases, whereas reference threshold 6 is correct on 2,204 cases. The effect size is
The improvement is therefore about 5.03 percentage points. A graph with a truncated axis can make that difference look large, but its numerical effect size does not change. Because the specified minimum is 10 percentage points, the initial hypothesis is not supported.
“Not supported” does not mean “the two rules are certainly identical.” The result shows a positive observed effect that is smaller than the target set in advance. This unit does not calculate a confidence interval or a p-value; do not attach statistical interpretations that have not been calculated.
10.9 Negative results are still results
After seeing 0.0503, we must not lower the effect threshold from 0.10 to 0.05 and claim that the original plan succeeded. Nor may we delete losing thresholds or display only favorable replications.
Negative results are useful because they:
- constrain plausible claims;
- prevent others from retracing a dead end without knowing its outcome;
- may reveal that a practical target is too ambitious; and
- help design the next experiment without rewriting the history of this one.
A new experiment may have a new hypothesis. What is dishonest is rewriting the old plan as though the new target had been specified from the start.
10.10 The canonical Unit 7 record
Run from the project root:
python source/code/unit07_experiment_design.py --output output/unit07-results.jsonThe JSON stores the question, hypothesis, grid, variables, controls, selection rule, stopping rule, seed schedule, each replication summary, development results, holdout results, exact effect size, negative result, and limits of the evidence. Fractions are stored as numerator and denominator and as decimals to four places. Keys are sorted, indentation and line endings are fixed, and core_sha256 binds the record before the hash field is added.
In the same Python environment, the same command produces the same bytes. The Python version is also recorded because the details of the pseudorandom generator form part of the environment that must be fixed.
10.11 Experiments and proof
This experiment establishes something very limited about execution: for the recorded seeds and code, the numbers of correct predictions can be recalculated. It provides empirical evidence about five thresholds on finite synthetic samples.
It does not prove that threshold 5 is best:
- outside the grid from 3 to 7;
- for every sequence of data;
- for other case generators; or
- for real phenomena not modeled by the generator.
A general proof requires a precise mathematical claim. For example, if the score distribution and outcome probabilities are defined exactly, one can calculate the expected accuracy of each threshold by summing probabilities over the ten scores. That calculation is an argument about the defined model, not a guarantee that the model represents the real world.
10.12 Exercises
10.12.1 Exercise 1 - variables and confounders
An experiment tests threshold 4 on the first 100 cases and threshold 6 on the next 100 cases. Identify the main variable, the measured outcome, and a possible confounder. Suggest one improvement.
Ask whether the two thresholds encounter equally difficult cases.
The main variable is the threshold, and the outcome may be the proportion of correct predictions. The group of cases is a confounder because the first and second sequences may differ in difficulty. The most direct improvement is to evaluate both thresholds on the same 200 cases. If that is not possible, randomize the assignment order and replicate according to rules fixed in advance.
10.12.2 Exercise 2 - a seed is not a time machine
Explain why saving a seed is sufficient to rerun the Unit 7 simulation but insufficient to repeat a temperature measurement in a laboratory.
Distinguish a pseudorandom sequence from physical disturbances and the state of the equipment.
The simulation calculates the entire sequence from an algorithm and a seed, so the same pair can generate the same cases in a fixed environment. Laboratory temperature is influenced by sensors, calibration, time, location, and physical disturbances that do not come from random.Random. A seed must be supplemented with raw data, equipment state, time, procedure, and independent replications.
10.12.3 Exercise 3 - selection and holdout
On development data, thresholds 4, 5, and 6 make 760, 810, and 790 correct predictions out of 1,000 cases. Which threshold is selected? Why must you not change the selection after seeing that threshold 6 is slightly better on the holdout?
Apply the selection rule to the development data, then preserve the holdout’s role as an untouched assessment.
Threshold 5 is selected because 810 is the largest number of correct predictions. If the selection changes to 6 after inspecting the holdout, those data have influenced selection and no longer provide a separate assessment. Threshold 6 can become a hypothesis for a new experiment with a new holdout, but the old experiment’s results must still be reported according to the original plan.
10.12.4 Exercise 4 - effect size and stopping rules
Calculate the improvement of threshold 5 over threshold 6 from 2,355 and 2,204 correct predictions in 3,000 cases. Is the target of 0.10 reached? May the experiment stop early at a replication showing 0.11?
Subtract the counts of correct predictions, then divide by the common number of cases.
The improvement is , or about 5.03 percentage points. The target of 10 points is not reached. The experiment must not stop at a single replication showing 0.11 because the original rule requires every replication; stopping based on the results would change the design and make the summary overly favorable.
10.12.5 Exercise 5 - negative results and limits of evidence
Write an honest conclusion for the Unit 7 experiment and one claim that must not be made. Then state what would be needed to prove a general claim about the best threshold in a defined model.
Separate the holdout effect, the target of 0.10, and the full space of possibilities.
An honest conclusion: threshold 5 was selected from the grid and then outperformed threshold 6 by about 5.03 percentage points on the holdout, but did not reach the specified minimum effect of 10 points. An invalid claim: threshold 5 has been proved best for all data or for the real world. For a general claim within an exact model, specify the score distribution and outcome probabilities, calculate the expected accuracy of every candidate (or of the entire threshold domain), and compare them using an argument that covers them all.
10.13 Summary
- The question, hypothesis, comparator, outcome measure, and minimum effect are specified before results are seen.
- Other variables are controlled so that differences are not confounded by sample difficulty or changes in procedure.
- The grid, tie-breaking rule, replications, seed schedule, and stopping rule are all part of the plan.
- A seed reproduces pseudorandomness, not unrecorded physical nondeterminism.
- Development data select parameters; the holdout assesses the selection once.
- Numerical effect size matters more than visual impression, and negative results must still be reported.
- Finite experiments provide limited empirical evidence; general proofs require arguments covering the domain of the claim.