Failure Policy, Retries, and Incomplete Outputs¶

Production operation is not only about getting a run to finish.

It is also about deciding what the workflow should do when something goes wrong:

retry
stop
keep evidence
rerun from incomplete state

Those decisions form a failure policy. If that policy is vague, the repository will recover inconsistently and leave ambiguous state behind.

How to read this page¶

Read this page with one pressure in mind:

when something fails, what exactly is the repository promising to trust, rebuild, or reject next?

That question keeps failure handling concrete. It is not mainly about convenience. It is about output trust.

Why this lesson matters early¶

Teams often add retries before they can classify failures. That is backwards.

If you do not know whether a failure is transient, semantic, or publication-related, any retry policy becomes guesswork. Module 03 needs the classification habit before it needs more flags.

The sentence to keep¶

When a job fails, ask:

should the next action be retry, rerun, or refusal to trust the output?

That question is the center of this page.

A strong beginner habit¶

Before changing retry behavior, write the failure class in plain language.

For example:

transient infrastructure interruption
bad semantic input or config
incomplete publication after interruption

If you cannot name the class, you are not ready to automate the response.

Retries are for transient failure, not semantic uncertainty¶

A retry is justified when the same declared job is likely to succeed on a second attempt without changing its meaning.

Typical examples:

network or mirror hiccups during a download step
scheduler or infrastructure instability
temporary shared-filesystem timing issues

A retry is not a fix for:

wrong inputs
bad parameters
nondeterministic rule logic
hidden semantic state

If the job meaning is wrong, retrying only repeats the wrong job.

The contrast that matters¶

The important contrast is not:

failed once
failed twice

The important contrast is:

failure could change if the same contract is attempted again
failure will repeat until the contract or inputs are repaired

Incomplete outputs are part of the contract¶

Production workflows must be honest about partially written outputs.

The repository should answer:

what happens if a job fails halfway through
whether any final-looking file remains behind
how the next run recognizes that state

This is why incomplete-output handling matters. It is not merely cleanup.

A quick self-check before you keep reading¶

Think about one failed run you have seen.

Could you say clearly:

whether the final output was trustworthy
whether the failure left recognizable incomplete state
whether rerunning would have been honest

If not, the repository probably had a weak failure story.

Keep the recovery story small and explicit¶

A healthy recovery story usually looks like this:

a job fails
logs preserve evidence
partial or incomplete outputs are not trusted as final
the next run reruns the affected work deliberately

That is a much better teaching model than "rerun until it works."

A weak first response¶

Weak production habit:

enable retries everywhere
keep partial outputs casually
assume later jobs will sort it out

This feels resilient. It is often the opposite:

poison artifacts survive longer
later failures become harder to interpret
maintainers lose the original failure boundary

The repository becomes noisier instead of safer.

A stronger failure-policy split¶

Use three categories:

1. Retryable failure¶

The job can be attempted again because the contract is still the same and the failure is likely transient.

2. Rerunnable incomplete state¶

The job produced incomplete state that should be recognized and rebuilt, not trusted.

3. Fail-fast contract error¶

The job or configuration is wrong in a way that no retry should hide.

This split gives the workflow one of the most important operational qualities: honest recovery.

Logs belong in the same discussion¶

Failure policy without logs is weak because the repository cannot explain what happened.

Per-job logs are especially important in production because they answer:

which exact job failed
what command it ran
whether the error looks transient or semantic

Logs do not replace recovery policy. They make it reviewable.

The common bad repair¶

A common bad repair is to raise retries until the workflow "usually passes."

That can hide the exact distinction this page is trying to teach:

a transient problem that deserves retry
a semantic defect that deserves repair
a publication defect that deserves rerun from incomplete state

One simple decision table¶

Situation	Better response
filesystem lag delayed visible outputs even though the job completed	tune latency or rerun policy, not workflow meaning
external infrastructure failed briefly	allow retry
wrong sample or bad config key caused the failure	fail fast and fix inputs
job left behind a partial final-looking output	treat it as incomplete and rerun deliberately
repeated retries still produce different outputs or errors	stop and inspect the rule contract

This is the kind of operational table a human team can actually use.

Add scheduler rejection to the failure model¶

A job can fail before any workflow command runs. The scheduler may reject a request because an account is invalid, a partition does not exist, or a required resource is absent. Retrying the same request does not make that contract valid.

flowchart TD
    F["run did not complete"] --> S{"scheduler accepted job?"}
    S -->|"no"| R["repair submission policy"]
    S -->|"yes"| C{"command started?"}
    C -->|"no"| I["inspect worker launch and environment"]
    C -->|"yes"| E{"transient execution failure?"}
    E -->|"yes"| T["bounded retry with preserved logs"]
    E -->|"no"| M["repair semantics, code, or data"]

This adds an important boundary to the retry question:

What changed between the failed attempt and the next attempt?

If the answer is "nothing," a retry is justified only when the source is known to be transient. Queue delay is transient. A missing memory declaration is not. A worker losing a network connection may be transient. An invalid threshold is not.

Rehearse four failure classes¶

Use the scheduler-policy audit:

cd programs/reproducible-research/deep-dive-snakemake/capstone
gmake scheduler-policy-audit

Classify the generated cases before reading their decisions:

Case	Failure boundary	Retry unchanged?	Repair
accepted mapping	none in local translation	not applicable	site integration next
missing memory	submission contract	no	declare and map memory
under-provisioned memory	policy translation	no	preserve the rule declaration
semantic leak	workflow meaning	no	move threshold to semantic config

All three rejected cases are deterministic. Running the adapter ten times would produce ten equivalent rejections. The correct response is to repair ownership, not to increase attempts.

Separate incomplete output from scheduler state¶

An accepted scheduler job may still die while writing an output. You then have two different records:

scheduler state says whether and where execution occurred;
filesystem state says whether a declared output exists and is complete.

Neither record substitutes for the other.

sequenceDiagram
    participant W as Workflow
    participant S as Scheduler
    participant J as Worker job
    participant O as Output store
    W->>S: submit recorded resource request
    S-->>W: job identifier
    S->>J: start allocation
    J->>O: write through temporary path
    alt command and validation succeed
        J->>O: publish declared output
        J-->>S: exit 0
    else command or validation fails
        J-->>S: nonzero exit
        O-->>W: no published final output
    end

The safe recovery story is:

preserve the scheduler job ID and worker log;
keep temporary artifacts outside the final output path;
determine whether the failure is transient or deterministic;
repair the cause when deterministic;
rerun through the declared Snakemake target;
verify output content, not merely file existence.

This is why rerun-incomplete is useful but insufficient. It can authorize rerunning an incomplete target. It cannot decide whether the cause was safe to repeat.

Write a retry budget that can be reviewed¶

Field	Example	Review purpose
eligible failure	transient object-store timeout	prevents semantic retries
maximum attempts	two retries	bounds wasted work
backoff	30 seconds, then 120 seconds	avoids repeated pressure
preserved evidence	attempt log and scheduler job ID	supports diagnosis
stop condition	same exit code twice	forces investigation
output handling	publish only after validation	prevents partial success

A profile may select a bounded retry mechanism. The repository still needs a durable explanation of these conditions. retries: 3 alone hides the most important reasoning.

What "keep evidence" should mean¶

Keeping evidence does not mean keeping every broken file forever.

It means:

preserve logs
make incomplete state recognizable
avoid promoting partial outputs into trusted boundaries

That is how the next maintainer can tell whether the workflow should retry, rerun, or be repaired.

What a good explanation sounds like¶

Strong:

The repository retries transient failures, reruns incomplete work deliberately, and fails fast on semantic defects because those three cases imply different trust decisions.

Weak:

If it flakes, we keep trying until it goes green.

Common failure modes¶

Failure mode	What it looks like	Better repair
retries enabled indiscriminately	wrong jobs get repeated instead of fixed	reserve retries for transient failure classes
partial outputs remain trusted	downstream steps read poison artifacts	publish atomically and rerun incomplete work deliberately
logs are global or missing	nobody can locate the failing job clearly	keep per-job logs or equivalently narrow evidence
incomplete handling is inconsistent across contexts	local recovery differs from CI without explanation	make the policy explicit in profiles and repository docs
retry is used to mask nondeterminism	failures seem random and irreproducible	repair the underlying rule contract first

The explanation a reviewer trusts¶

Strong explanation:

this rule may be retried for transient infrastructure errors, but incomplete outputs are never trusted as final; failed jobs keep per-job logs, and the next run reruns incomplete work instead of silently continuing from poison artifacts.

Weak explanation:

if it flakes, we just retry and usually it settles down.

The strong version gives an operational contract. The weak version gives a coping habit.

End-of-page checkpoint¶

Before leaving this page, you should be able to:

name one failure that deserves retry and one that does not
explain why incomplete-output handling is part of output trust
describe how logs support recovery decisions
explain why retries cannot repair semantic workflow defects