15  A Verifiable Capstone Project

15.1 Learning objectives

After completing this unit, you will be able to:

  • turn a mathematical question into an experiment with explicit limits on its claims;
  • separate mathematical claims, implementation claims, and empirical observations;
  • submit source, data, environments, commands, tests, and artifacts linked by hashes;
  • ask someone else to reproduce the results without additional spoken instructions;
  • respond to substantive review with verifiable changes or reasons; and
  • explain which parts of a project constitute proof and which provide only computational evidence.

Local prerequisites: all O002 core units relevant to the project, and all mathematical prerequisites of the chosen topic. A project must not use an advanced mathematical topic merely because a library can compute with it.

15.2 The question comes before the tools

A project starts with a question, not a desire to use a particular library. A good question specifies the objects, domain, quantities to observe, and the form of an answer that would be meaningful. For example:

How does the error in evaluating an algebraic expression change as the input approaches a point of cancellation, and which reformulation preserves more significant digits over the specified range?

This question does not promise a universal theorem. It defines an experimental scope, a comparison of methods, and an error measure that can be audited.

15.3 Three kinds of claim

Label each project conclusion with one of the following kinds.

Kind Example Evidence required
mathematical two formulas are equivalent on a specified domain a symbolic argument stating the domain and exceptions
implementation a function returns a residual within a specified bound inspection, tests, and edge cases
empirical method A is more accurate on a particular test grid data, parameters, environment, error measure, and analysis

The most serious error is to promote an empirical observation to a mathematical claim without an argument. Another error is to give a correct mathematical proof without checking whether the program actually implements the method it justifies.

15.4 The minimum reproducibility package

A minimal project package has the following structure.

project/
  README.md
  LICENSES/
  environment/
  source/
  tests/
  inputs/
  outputs/
  RUN_MANIFEST.json

RUN_MANIFEST.json records the schema version; command as an array of arguments; environment with the runtime, runtime version, and dependencies; the parameters object; lists of inputs and outputs; size and hash records in artifacts; and an array of claims. Every input and output path must refer to a registered artifact. Each claim object contains id, kind, statement, evidence, and limitations, and its evidence must refer to registered outputs. Artifact hashes thus also bind the evidence supporting each claim. The manifest must contain at least one artifact, one output, and one claim, and paths must not be repeated. Paths must use canonical POSIX form and be relative to the package root; machine-specific user-profile paths must not be included in the public package.

This unit’s verification script takes a manifest and a package root:

python source/code/unit12_capstone.py verify

This short command uses the default layout above: package project, manifest project/RUN_MANIFEST.json, and report output/unit12-results.json. The --root, --manifest, and --output options are available for other layouts.

The verifier does not determine whether a mathematical argument is correct. It checks that the named files exist, their sizes and hashes match, paths stay inside the package, the execution contract has the minimum required structure, claims contain the required fields, and each claim’s evidence refers to outputs that have actually been verified. It does not execute command, assess the contents of the README or licenses, or establish that the minimum folder structure is sufficient; those matters are still checked through blind reproduction and the rubric.

15.5 A bounded blind reproduction

Give the package to a reviewer who did not help create it. The reviewer follows the README and records:

  1. the environment actually installed;
  2. the commands executed;
  3. input and output hashes;
  4. which tests passed or failed;
  5. differences in results; and
  6. ambiguous or incomplete instructions.

“It worked on the author’s computer” is not a reproducibility standard. Conversely, differences between machines do not necessarily mean the research is wrong. Trace differences to versions, platforms, nondeterminism, tolerances, or defects in the package.

15.6 Review and revision

Substantive review challenges claims, methods, domains, tests, visualizations, or reproducibility. The project’s response includes:

  • a summary of the issue raised;
  • a decision to accept, partly accept, or reject the criticism;
  • the changes made to files or arguments;
  • any new evidence added; and
  • the limitations that remain.

Rejecting a comment is allowed, but “my computer gives the same result” is not a sufficient reason when the comment concerns the validity of a claim.

15.7 Completion rubric

A project passes only when all of the following conditions are met:

  1. the question and domain are explicit;
  2. sources, data, and component rights are clear;
  3. the environment can be installed and the commands rerun;
  4. tests cover relevant invariants and edge cases;
  5. artifacts and the manifest agree;
  6. visualizations include labels, units, scales, and descriptions;
  7. mathematical, implementation, and empirical claims are kept distinct;
  8. conclusions state their limitations;
  9. a bounded blind reproduction is documented; and
  10. one response to substantive review is included.

Failure to meet one condition does not necessarily invalidate the entire investigation, but the project does not meet the O002 standard until the shortcoming is corrected or its resolution is openly documented.

15.8 Exercises

15.8.1 Exercise 1 - classifying claims

Classify the following statement: “On Python 3.13 under Windows, algorithm A takes less time than B for all 100 sizes tested.” Does the statement prove that A is always faster?

Consider the platform, version, and set of sizes.

It is an empirical observation limited to that environment and the 100 test sizes. It does not prove that A is always faster for every size, dataset, platform, version, or implementation. The conclusion must retain those limits.

15.8.2 Exercise 2 - an incomplete manifest

A manifest records the script name and output image, but no parameters, library versions, or data hashes. Name three reproducibility failures that could arise.

Ask whether the reviewer can identify the same inputs, environment, and bytes.

The reviewer might use different parameters, encounter different behavior because of library versions, or use data with the same filename but different bytes. All three situations can produce a different image without a reliable way to identify the source of the difference.

15.8.3 Exercise 3 - an unsafe path

Why must a public manifest not refer to a user-profile path such as user-profile/Name/Desktop/data.csv? Provide a replacement.

Consider privacy and portability.

The path exposes the local profile structure and does not exist on the reviewer’s machine. Store the data inside the package, for example at inputs/data.csv, then use a path relative to the project root and bind the file’s bytes with its size and SHA-256 hash.

15.8.4 Exercise 4 - nonidentical reproduction

Two machines produce values that differ in the last digit, but both fall within the tolerance specified before the experiment. Has reproduction failed?

Distinguish semantic reproduction from byte identity.

Not necessarily. If the contract permits floating-point variation and both values meet a justified tolerance, the semantic results can agree even when their numerical bytes differ. The manifest must still record the platform and versions, however, and artifacts claimed to be byte-identical must be checked with hashes.

15.8.5 Exercise 5 - responding to review

A reviewer points out that a graph truncates the vertical axis, making small differences appear very large. Write an adequate response.

Explain the decision, changes, and their effect on the conclusion.

Acknowledge that the axis choice can exaggerate the visual impression. Add a graph showing the full range or a clear truncation marker, retain the truncated graph only if it helps reveal local variation, and report the effect size numerically. Revise the conclusion if the previous claim depended on a misleading visual impression, then record the new artifact’s hash.

15.9 Summary

  • A project starts with a question and a domain, not with tools.
  • Mathematical, implementation, and empirical claims require different evidence.
  • Manifests and hashes bind artifacts, but do not prove theorems.
  • Blind reproduction tests whether the package and instructions are sufficient.
  • Substantive review must receive a response containing verifiable changes or reasons.
  • Completing a project requires mathematical, computational, documentary, and collaborative work, all of which must be open to inspection.