13  Automation and Reproducible Workflows

13.1 Learning objectives

After completing this unit, you will be able to:

  • describe an experiment as a graph of tasks and dependencies;
  • distinguish sources, intermediate products, and final artifacts;
  • rebuild only when recorded inputs, parameters, code, or environment change;
  • detect missing dependencies and cycles before running tasks;
  • write artifacts atomically so that failures do not leave apparently complete results;
  • explain why timestamps alone are not evidence of identical contents; and
  • audit workflow records without treating a successful process as mathematical proof.

Local prerequisites: Units 1–2 and 8–9. You should understand functions, files, hashes, configuration, and tests before automating them.

13.2 From a command list to a graph

A small experiment is often written as a list of commands:

generate-data
analyze-data
draw-figure
write-report

That sequence hides the reason for each step. A dependency graph states that the analysis needs the data, the figure needs the analysis, and the report needs both the analysis and the figure. With that information, a workflow manager can determine a valid order and which work needs to be repeated.

The graph must be acyclic. If analysis requires report while report requires analysis, there is no first task that can be completed.

13.3 Sources and artifacts

  • Sources are written or obtained outside the workflow: code, configuration, raw data, and licenses.
  • Intermediate products can be regenerated but support the next step.
  • Final artifacts are delivered to readers or reviewers.

Do not make a final artifact the only copy of source data. An intermediate product that is expensive to regenerate may be cached, but its hash and the rules for creating it must still be known.

13.4 When a task is up to date

Timestamps are useful for local optimization, but they can change when files are copied or machine clocks disagree. A robust record binds together:

  1. the hash of each input’s bytes;
  2. the hash or version of the task code;
  3. canonicalized parameters;
  4. the relevant environment identity; and
  5. the hash of each output.

The Unit 10 script stores the complete, canonical fingerprint preimage: input names and their SHA-256 hashes, the task code’s SHA-256 hash, parameters, and the identity of the runtime actually running the example. The runtime identity includes the Python implementation and version, cache tag, operating-system family, machine architecture, and byte order. It deliberately excludes the hostname and profile paths. task_fingerprint is the SHA-256 hash of the preimage’s canonical JSON bytes, so it can be recalculated directly from the record.

If any input changes, the task and all its descendants become out of date. If everything matches and the outputs still have their recorded hashes, the task can safely be skipped. The checker also accepts a list of required output names. An empty output mapping, a missing required output, or an extra undeclared output makes the task out of date. This unit does not define a contract for tasks with no outputs.

13.5 Atomic writes and failure

Writing directly to results.json can leave a truncated file if the process stops. A safer pattern is:

  1. write to a new temporary file in the same directory;
  2. close it and synchronize it if crash durability is required;
  3. verify its structure and hash;
  4. atomically rename the temporary file to the destination name; and
  5. do not remove the old result until its replacement is ready.

The temporary name must be unique and must not refer to an existing link. Cleaning up temporary files is local failure handling, not a replacement for a manifest.

13.6 Named targets and checks

The O002 workflow has at least these targets:

test     run tests without building the reader
results  produce experiment artifacts
html     build the web reader after test and results
pdf      build the print reader after test and results
qa       check both readers and the manifest
all      run the complete set of acceptance checks

The all target must not publish. Publication is a separate transaction that uses only artifacts that have passed qa.

13.7 An example workflow

This unit’s script validates a small graph, calculates a topological order, and creates a canonical record containing the fingerprint preimage, actual runtime, and output contract:

python source/code/unit10_pipeline.py --output output/unit10-results.json

The example does not execute shell commands. It models workflow decisions so that ordering, cycle detection, and freshness logic can be tested without external effects.

13.8 Automation does not determine truth

A green workflow establishes that the programmed checks passed in that environment. It does not prove that the checks are sufficient, the model is appropriate, or the theorem is true. Automation reduces recurring oversights; mathematical responsibility remains with the designers and reviewers.

13.9 Exercises

13.9.1 Exercise 1 — task order

Task data has no dependencies; fit requires data; plot requires data and fit; report requires plot. Give one valid order.

Every task must appear after all its dependencies.

The order data, fit, plot, report is valid. plot cannot precede fit, and report cannot precede plot. When there are several independent tasks, more than one order may be valid.

13.9.2 Exercise 2 — a cycle

Explain why the dependencies a -> b, b -> c, and c -> a must be rejected before any task runs.

Look for a task whose dependencies have all been completed.

There is no initial task: a waits for b, b waits for c, and c waits for a. Running part of the workflow before detecting the cycle can leave apparently valid artifacts, so graph validation must precede any side effects.

13.9.3 Exercise 3 — timestamps

A data file is copied with the same contents but a new timestamp. Must the analysis always be repeated? Compare timestamp-based and hash-based policies.

A timestamp records a filesystem event, not contents.

A timestamp-based policy may rebuild even when the contents are unchanged. A hash-based policy can skip the analysis if the data bytes, code, parameters, environment, and recorded outputs all still match. Hashing requires more data to be read but binds more directly to the contents.

13.9.4 Exercise 4 — a write failure

Why is writing new output directly over an old artifact dangerous? Set out a safe writing sequence.

Consider a process that stops after writing half the bytes.

A failure can destroy the old artifact and leave truncated output under the official filename. Write to a uniquely named temporary file in the same directory, close and verify it, then replace the destination atomically. If a step fails, the old artifact remains available and the temporary file can be disposed of.

13.9.5 Exercise 5 — a green workflow

All workflow targets pass. Give two reasons why the mathematical conclusion could still be wrong.

Tests may be incomplete, and the model may not match the question.

The tests may omit a case that exposes an incorrect implementation, and the model or mathematical assumptions may be wrong even when the code implements them accurately. A green workflow strengthens the reproducibility of the checks, not their logical completeness.

13.10 Summary

  • A dependency graph explains ordering and the effects of changes.
  • A complete fingerprint preimage makes the hash of inputs, code, parameters, and runtime reproducible.
  • Content hashes and output contracts bind freshness more directly than timestamps; an empty output mapping is not considered up to date.
  • Cycles and missing dependencies must be rejected before side effects occur.
  • Atomic writes prevent half-finished outputs from using the official filename.
  • Automation makes checks consistent but does not replace mathematical judgment.