12  Data, Configuration, and Provenance

12.1 Learning objectives

After completing this unit, you will be able to:

  • distinguish raw data, derived data, and run reports;
  • specify schemas, types, column meanings, and units;
  • separate variable configuration from transformation code;
  • record environment identity and the limits of lockfiles;
  • use relative paths that cannot escape the project root;
  • record the origins and rights status of each component;
  • bind artifacts to byte counts and SHA-256 hashes without overstating what hashes establish;
  • exclude private data from public artifacts; and
  • distinguish semantic reproducibility from byte-for-byte reproducibility.

Local prerequisites: Units 1–4 and 7–8, tables, ratios, units, text files, and simple functions. The basic route uses only the Python standard library.

12.2 Three data layers

Raw data are the records received before this unit’s analytical transformations. “Raw” does not mean without a history: a sensor, form, or export process may already have changed them. The term simply identifies the frozen entry point for this run.

Unit 9 uses five synthetic records:

sample_id length_cm mass_g
S01 10.0 20.0
S02 12.0 24.6
S03 8.0 15.8
S04 14.0 28.7
S05 11.0 21.8

Derived data are produced by the rule

r=mass in gramslength in centimeters, r=\frac{\text{mass in grams}}{\text{length in centimeters}},

then rounded to three decimal places using the round half even rule. The derived column is named mass_per_length_g_per_cm.

A run manifest is not new measurement data. It records which files were used, which transformation was performed, which configuration and environment applied, and which output bytes were produced.

Do not overwrite raw data with derived results. If a transformation needs to be corrected, retaining the raw data with recorded hashes allows the outputs to be regenerated and compared.

12.3 Schemas and units

A schema is a structural contract. This unit’s raw-data schema specifies:

Field Type Unit Meaning
sample_id text of the form S + two digits none synthetic identifier
length_cm positive decimal text cm length
mass_g positive decimal text g mass

Storing decimals as text avoids an initial conversion to floating point. The code rejects numeric values that are not text, then converts the validated text to Decimal for calculation. An identifier has no unit; None is more accurate than an empty unit field that could mean “forgotten.”

The name length without a unit is insufficient. A value of 10 could mean 10 mm, 10 cm, or 10 m, and these give different ratios. Units must stay with the data through to the output; the derived-data schema specifies g/cm.

Validation checks field names, identifier format, uniqueness, finiteness, and positivity. Passing schema validation does not guarantee correct measurements, but it prevents several structural misinterpretations.

12.4 Configuration separate from code

Transformation code states how the ratio is calculated. Configuration states the run settings that may legitimately vary, for example:

{
  "derived_path": "data/derived/unit09_mass-per-length.csv",
  "ratio_decimal_places": 3,
  "raw_path": "data/raw/unit09_measurements.csv",
  "rounding": "ROUND_HALF_EVEN",
  "schema": "o002.unit09.config.v1"
}

If the number of decimal places is embedded in many lines of code, a small change can easily become inconsistent. A single validated configuration makes that choice inspectable and allows it to be bound to a hash.

Separating configuration does not mean accepting arbitrary values. This unit accepts only 0 through 9 decimal places and one tested rounding rule. Configuration is input with a schema, not a back door for executing text as code.

12.5 Environment identity and locks

Outputs can change even when the data and code appear identical, because language, library, operating-system, or hardware versions differ. The Unit 9 manifest records the Python implementation and version and states that the basic route uses only the standard library.

Because there are no external dependencies, this unit’s “environment lock” is small: the Python identity and the Decimal arithmetic context are serialized canonically and bound to a SHA-256 hash. The context fixes precision at 28 digits, round-half-even rounding, exponent limits, and operation traps; it does not inherit the calling process’s precision or traps. Projects using NumPy, SymPy, SageMath, or other libraries need a lockfile that specifies exact versions and package sources. A list of names without versions is not a lock.

A lockfile does not freeze the entire world, either. Differences in operating system, locale, CPU, or system libraries may still matter. Record the layers that affect the question, and do not call environments “identical” merely because one version number matches.

12.6 Safe relative paths

The manifest stores logical paths such as data/raw/unit09_measurements.csv, not user-profile/Name/Desktop/data.csv. Relative paths can move with the project and do not disclose local account names.

The unit’s validator accepts canonical relative POSIX paths. It rejects:

  • /etc/secret or C:/secret, because they are absolute;
  • ../secret and data/../secret, because they can escape the root or obscure their destination;
  • backslashes, so one form is used on every system; and
  • redundant forms such as data//raw/file.csv.

Each artifact role must also have a unique path after Unicode normalization and case folding. Consequently, data/raw.csv and DATA/RAW.CSV are treated as a collision, so that the same manifest remains safe on case-insensitive filesystems.

Path validation is a security and reproducibility boundary. It is not, however, permission to read every file that happens to be inside the project. The list of permitted inputs must still be limited by the run plan.

12.7 Provenance and component-specific rights

Provenance answers “where did this component come from, and what happened to it?” Rights answer “on what basis may this component be used or distributed?” The two are related but distinct.

The manifest records separately:

  • synthetic raw data defined in the code;
  • derived data and their transformation formula;
  • configuration;
  • environment identity; and
  • the Unit 9 code, which points to LICENSE-CODE.md and the MIT license.

The example data contain no third-party material. These records are not used to make a blanket licensing claim about every project file. If a CC BY-SA image, MIT code, and specially licensed data are combined, each component still needs its own creator/source, license, change record, and restrictions. Provenance must not be discarded merely because the derived results are numbers.

12.8 SHA-256 and byte counts

For each component, the manifest stores a logical path, media type, role, byte count, and SHA-256 hash. If one byte changes, the hash will almost certainly change. The size helps detect empty files or accidental truncation and provides a simple check before hashing.

A hash answers a question about byte identity. It does not prove that:

  • the transformation formula is correct;
  • the units are appropriate;
  • the data were collected ethically;
  • the recorded license is valid; or
  • the manifest includes every component it should.

Those questions require other checks. Matching hashes of an incorrect file only establish that two people have the same incorrect file.

12.9 Privacy and exclusions

Reproducibility is not a reason to publish names, email addresses, phone numbers, precise locations, or free text that could identify someone. The unit’s dataset uses S01 through S05 as synthetic identifiers and explicitly rejects private fields.

In real research, strategies may include:

  • separating restricted data from public artifacts;
  • publishing schemas, code, synthetic data, and safe summaries;
  • recording how authorized researchers can apply for access;
  • removing unnecessary fields before analysis; and
  • checking whether combinations of fields can still identify people.

Writing a confidential dataset’s hash into a public manifest can also leak information in some circumstances, especially when the space of possible values is small. Decide what may be published before creating a public manifest.

12.10 A canonical run manifest

Run from the project root:

python source/code/unit09_provenance.py --output output/unit09-manifest.json

The script does not access the network or the current time. The manifest contains raw and derived records, schemas, configuration, environment identity, artifact hash bindings, the input-code-configuration-output chain, privacy policy, component-specific rights, and limits on reproducibility. The JSON uses UTF-8, sorted keys, fixed indentation, and a single final LF newline. core_sha256 binds the contents before that hash field is added.

The raw and derived CSV files in this example are logical artifacts whose canonical bytes are constructed in memory and bound to hashes in the manifest. The command therefore writes only the requested JSON, but you can reconstruct the exact CSV files from the stored records and column rules.

12.11 Semantics and bytes

Two outputs are byte-for-byte reproducible if every byte matches. They necessarily have the same SHA-256 hash. This criterion is useful for checking deterministic workflows, archives, and distribution.

Two outputs can be semantically reproducible even when their bytes differ. For example, the following JSON texts store the same mapping:

{"a":1,"b":2}
{
  "b": 2,
  "a": 1
}

Whitespace and key order differ, so the hashes differ, but a JSON parser produces the same value. In numerical computing, two algorithms can also produce equivalent values within a specified tolerance and with specified units without producing identical bytes.

A report must therefore state the level required. Unit 9’s deterministic repetition targets identical bytes in a locked environment. Broader scientific validation must also compare schemas, units, formulas, and the meaning of the results.

12.12 Exercises

12.12.1 Exercise 1 — raw data, derived data, and lineage

If mass_g for S03 changes from 15.8 to 15.9, which components must have new hashes, and which should remain unchanged?

Follow the chain raw data → transformation → derived data; the code and configuration are not edited.

The raw-data hash changes, although its byte count may remain the same, and the derived-data hash must change because S03’s ratio is calculated from the new mass. The manifest’s core_sha256 also changes because its artifact bindings change. The code, configuration, and environment-identity hashes remain the same if their bytes are unchanged. Provenance must record that the new outputs came from that revision of the raw data.

12.12.2 Exercise 2 — schemas and units

A table replaces length_cm with a column named length without updating the schema. Why is the value 10.0 insufficient for calculating an interpretable ratio? Give a correction.

Numbers do not automatically carry units.

10.0 could mean millimeters, centimeters, meters, or another unit. The ratio and its units change with that choice. Correct this by using an unambiguous field such as length_cm, stating unit: cm in the schema, validating the data against the schema, and changing the output formula if unit conversion is required.

12.12.3 Exercise 3 — configuration and safe paths

A configuration contains raw_path: "../../private.csv", and the number of decimal places is written directly into three different functions. Identify two problems and give their corrections.

Check for directory traversal and for a single source of run settings.

The path ../../private.csv can escape the project root and must be rejected; use a canonical relative path from the permitted input list. Embedding the number of decimal places in three locations makes inconsistent changes easy. Store one ratio_decimal_places value in versioned configuration, validate its range, and have every call read the same value.

12.12.4 Exercise 4 — rights and privacy

You receive a third-party table containing measurements, names, and email addresses. State what must be separated or recorded before creating a public artifact.

Provenance, permission to redistribute, and analytical need are three different questions.

Record the source, creator, version, license or basis for permission, and changes to the table component. Separate names and email addresses from the public data unless there is a valid basis, need, and consent; use safe identifiers or synthetic data where appropriate. Check whether combined measurements can still identify people. Do not assume that the project’s code license automatically covers third-party data.

12.12.5 Exercise 5 — semantics, bytes, and hashes

Two JSON files have different key orders and whitespace but produce the same object when parsed. What can you say about their semantic reproducibility, byte-for-byte reproducibility, and SHA-256 hashes?

Compare the parsed value with the original sequence of characters.

They are semantically reproducible for that JSON mapping because their parsed values match. They are not byte-for-byte reproducible because their character sequences differ, so their SHA-256 hashes almost certainly differ. Each hash binds only its file’s bytes; to require the same hash, serialize both using the same canonical rules.

12.13 Summary

  • Raw data, derived data, and run manifests have different roles and must not overwrite one another.
  • Schemas include types, meanings, and units; structural validation is not proof of measurement quality.
  • Configuration is separated from code but still has a schema and limits.
  • Python identity and versioned dependencies form a minimum environment lock.
  • Canonical relative paths improve portability and prevent directory traversal.
  • Each component retains its own provenance and rights status.
  • Sizes and SHA-256 hashes bind bytes, not correctness, ethics, or licensing.
  • Privacy may require data to be excluded from public artifacts.
  • Semantic reproducibility does not always require identical bytes; a manifest must state its target level.