Module 03: Determinism, Debugging, and Self-Testing¶

Modules 01 and 02 teach truth and parallel safety. Module 03 asks what happens after the build starts living in the real world:

Does the graph stay stable when files change, machines differ, CI runs cold, and the team needs explanations instead of guesses?

This module is about keeping the build trustworthy under change. That means deterministic discovery, disciplined debugging, a stable CI-facing target surface, and selftests that prove the build system rather than simply exercising the binary.

The module promise¶

At the end of this session, you will not merely have run a deterministic build. You will have a review packet that explains:

which repository state defines the graph;
which observation explains one rebuild decision;
which public target CI is entitled to trust;
which selftest claim accepted a healthy build;
which controlled defect the same selftest rejected;
what the evidence still does not establish.

Those six items turn “the build seems stable” into a claim another maintainer can review.

flowchart LR
  state["semantic repository state"] --> graph["evaluated graph"]
  graph --> decision["rebuild decisions"]
  decision --> execution["recipe execution"]
  execution --> artifacts["declared artifacts"]
  artifacts --> review["bounded review decision"]

The arrows are also diagnostic boundaries. A changed artifact may come from changed discovery, a changed rule, a changed execution environment, or a faulty writer. Do not skip directly from the symptom to a guessed repair.

Why this day matters¶

A build can seem healthy for a long time while still carrying silent variability. Module 03 teaches you to stop calling that "flaky" in the abstract and start naming the exact fact that changed.

What this module is for¶

By the end of Module 03, you should be able to explain four things clearly:

why the same repository can behave differently across runs or machines
how to debug a rebuild with Make-native evidence instead of folklore
which targets are real public contracts for CI and which are only local conveniences
how to test the build system itself for convergence and equivalence

Entry diagnostic¶

Begin with the first row you cannot defend:

Claim	Evidence you should already know how to obtain	Return route
a target is stale for an explicit reason	one target/prerequisite edge and timestamps	Module 01
two independent prerequisites may run concurrently	the relevant subgraph and writer ownership	Module 02
discovery gives the same ordered membership	captured evaluated source list	this module
a rebuild has a Make-native explanation	bounded `--trace` evidence	this module
CI invokes a stable public contract	target promise and failure semantics	this module
the build rejects a controlled graph defect	failed selftest with preserved evidence	this module

If either of the first two rows is uncertain, revisit that earlier lesson before continuing. Module 03 observes and tests graph truth; it cannot substitute for it.

The live demo for this module¶

Use this sequence as the anchor for the day:

flowchart LR
  build["Build once"] --> repeat["Build again and expect no work"]
  repeat --> drift["Introduce one unstable input or discovery step"]
  drift --> trace["Trace the resulting rebuild or mismatch"]
  trace --> selftest["Write a selftest that catches it automatically"]

That pattern appears again and again in real repositories. The goal is to make it boring enough that you can do it under pressure.

Debug one claim at a time¶

“The build is nondeterministic” is too large to investigate. Replace it with a claim you can falsify:

the discovered source list changes order
the same declared inputs produce different bytes
a second successful run still executes recipes
-j1 and -j8 publish different artifact sets
CI calls a target whose meaning differs from the local target

Use the narrowest observation for the claim:

Claim	First observation
source membership or order changed	print the rooted, sorted source variable
a target rebuilt unexpectedly	`make --trace TARGET`
the graph does not converge	build, then `make -q TARGET; echo $?`
concurrency changes results	compare sorted checksum manifests
CI and local behavior differ	compare the exact public target and inputs

This table is a debugging discipline, not a list of magic commands. If the first observation does not answer the question, record what it ruled out before escalating.

Plan for the day¶

Block	Activity	Evidence you should produce
Determinism	compare unsorted shell discovery with canonical discovery	two captured source lists and an explanation
Debugging	predict and trace one rebuild and one no-op run	exact trace lines tied to graph facts
Interface	inventory targets consumed by humans and CI	a public target table with promises
Self-testing	test convergence and serial/parallel equivalence	a selftest that fails on a deliberate defect
Abstraction	inspect the optional `eval` example	proof that the core build works with it disabled
Practice	complete all ten exercises	evidence from both positive and negative cases

Suggested independent-study timing¶

Work	Approximate time	Stop when
entry diagnostic and discovery lesson	45 minutes	you can distinguish membership, order, and byte identity
forensic debugging	50 minutes	one rebuild explanation names target, cause, and trace evidence
CI contract and selftest design	75 minutes	every public claim has an oracle and a rejection case
abstraction and worked simulator	55 minutes	explicit and generated rules can be compared without guessing
exercises and answer review	90 minutes	the evidence packet survives review without terminal history

The times are planning aids, not completion criteria. Preserve your stopping state under artifacts/learning/deep-dive-make/module-03/ and resume from the first unsupported claim.

Do not treat deterministic output as “the command succeeded twice.” Compare the declared artifacts or their checksums.

Likewise, do not treat a passing product test as proof that the build graph is correct. A stale binary can pass its tests. Build-system tests must manipulate inputs and inspect rebuild, no-op, failure, and concurrency behavior.

A question to keep asking¶

Whenever someone says "the build is flaky," translate it into a smaller question:

Which input changed?
Which output changed?
Which target rebuilt or failed to rebuild?
Which proof route can show that directly?

That translation is one of the most valuable skills in the entire course.

Keep the evidence layers separate¶

Layer	Question	Typical evidence
discovery	which inputs became graph members, in what order	printed evaluated variable or database excerpt
selection	why did Make select this rule or target	bounded trace line
execution	what command ran and how did it exit	recipe log
artifact	what was actually published	path inventory, size, digest, semantic check
convergence	does the unchanged request require more work	query exit and no-op trace
rejection	can the proof detect a known violation	controlled defect and failed report

A dry run belongs to selection and predicted execution. It is not artifact evidence. A successful recipe belongs to execution. It does not prove convergence. Keeping these layers separate prevents accidental overclaiming throughout the module.

Study route¶

flowchart TD
  start["Overview"] --> core1["Determinism and Stable Discovery"]
  core1 --> core2["Forensic Debugging with Make Evidence"]
  core2 --> core3["CI Targets as a Public Contract"]
  core3 --> core4["Build-System Selftests"]
  core4 --> core5["Macros and Quarantined eval"]
  core5 --> example["Worked Example: Production Simulator"]
  example --> practice["Exercises"]
  practice --> answers["Exercise Answers"]
  answers --> glossary["Glossary"]

Read the module in that order the first time. When you come back later, go straight to the file that matches the failure or design decision you are facing.

The ten files in this module¶

How to use the file set¶

If you need to...	Start here
stabilize discovery and generation under repository change	Determinism and Stable Discovery
explain one rebuild with evidence instead of folklore	Forensic Debugging with Make Evidence
define which targets CI is actually allowed to trust	CI Targets as a Public Contract
prove the build system rather than only the program	Build-System Selftests
keep macros and optional `eval` under control	Macros and Quarantined Eval
see the whole module in one simulator	Worked Example: Production Simulator
test your own understanding	Exercises
compare your reasoning against a reference answer	Exercise Answers
stabilize vocabulary while reading the module	Glossary

The running example¶

This module uses the course capstone/ as a production simulator. It extends the earlier small builds with:

dynamic source discovery under src/dynamic/
generated headers
modeled hidden inputs
a selftest harness that checks convergence, compares declared serial/parallel inventories, and preserves reviewable failures
an optional eval surface that stays quarantined from the core build

That gives you one build that is intentionally close to production pressure while still small enough to reason about.

What a strong Module 03 day looks like¶

By the end of the day, you should have:

demonstrated one converged no-op build
created one deterministic-discovery failure or variability risk
explained one rebuild with a specific trace line
written or reviewed one selftest that fails on a deliberate defect
separated one CI-facing contract from one local convenience target

Package those results as:

module-03/
├── claim-table.md
├── discovery.txt
├── rebuild-note.md
├── ci-contract.md
├── accepted-selftest/
└── rejected-selftest/

The exact filenames are not graded. The separation is: a reviewer should not have to infer which evidence supports which claim.

Recovery route after a missed class¶

If you are studying without the session:

run the entry diagnostic;
read the determinism and forensic-debugging lessons;
complete Exercises 1, 2, and 7;
read the CI and selftest lessons;
run the worked example through its rejected case;
complete the remaining exercises;
compare answers only after writing your own decision and limit.

Do not start with the selftest report and reverse-engineer the concepts from its field names. The report becomes useful after you can distinguish graph, execution, artifact, and rejection evidence.

The central review question¶

Carry this question through the whole module:

If the build behaves differently tomorrow, on CI, or on another machine, what exact fact changed in the graph or its inputs?

Good Module 03 answers usually mention one or more of these:

unstable discovery
unmodeled hidden inputs
generator publication problems
a public target whose meaning drifted
a selftest that was too weak to catch the regression

Commands to keep open¶

These commands form the evidence loop for Module 03:

make help
make all
make test
make selftest
make --trace all
make -p

Use them constantly. This module is not complete when the build merely succeeds. It is complete when you can defend why it succeeds, when it should rebuild, and what proof the build gives about itself.

Learning outcomes¶

By the end of this module, you should be able to:

keep discovery and generation deterministic under repository change
explain rebuilds with --trace, -n, and -p instead of guesswork
define a stable CI-facing target surface and protect it from semantic drift
design selftests that prove convergence and serial/parallel equivalence
distinguish failed, passed, and not-run selftest boundaries from saved evidence
use Make macros and optional eval without making the graph opaque

Exit standard¶

Do not move on until all of these are true:

you can point to one cause of nondeterminism and repair it
you can quote the trace line that explains a rebuild
you can say which targets are part of the CI contract and why
you can run a selftest that fails meaningfully when hidden variability is introduced
you can explain why the optional eval surface does not control the core build
you completed ten exercises and can separate product tests from build-system tests

When those become ordinary, Module 03 has done its job.