Building Ratchet Runtime: What a Safe Restart Has to Prove

Anchor gave deterministic code control of an agent workflow’s order. My second draft covers the failures that changed my view of what this proved. This note is the technical half: what I built after deciding that an unattended restart needed its own explicit contract.

Ratchet Runtime is the small Python reference implementation I built for ordered workflows in which an agent may perform a step and a step may change an external system. I kept its boundary narrow: it is not a scheduler, agent framework or judge of the agent’s work. It owns the mechanics that must remain true when a process overlaps another run, dies between writes or returns a confident but unverified success.

I publish it as an educational and experimental alpha for local use, not a production or distributed runtime.

The boundary is the design

I drew Ratchet’s boundary after a scheduler has requested a run and a workflow has declared its steps. The runner acquires the right to update its state. For a side-effecting step, it records an intent before the external action. It then invokes the step, reads the result back through a runner-controlled check, and commits completion only after every check holds.

The separation matters because those components fail differently. A scheduler can fail before a run starts. An agent can report success without establishing it. A process can die after changing an external system. A second run can arrive while the first one is still alive. No single lock or retry policy covers all four.

Six guarantees, six different questions

Ratchet’s contract separates six guarantees instead of treating “deterministic” as one broad property.

Guarantee	The question it answers	Mechanism
Ordering	May this step run yet?	Declared predecessors must complete before the step starts
Exclusive runner tenure	Which run may update runner state?	A heartbeating, expiring ownership record with ownership-checked release
Position integrity	Which source progress is safe to retain?	Per-source positions remain pending until verified workflow completion
Verified completion	Did the step establish a mechanical fact?	Every step has a post-condition evaluated by the runner rather than the agent
Effect integrity	Is an external action safe to retry?	Intent before effect, a stable key, and read-back after interruption
State integrity	Which result is authoritative after a crash?	Durable file replacement and one completion record containing outcome and positions

I use tenure to mean the temporary, exclusive right to mutate runner state. It is renewed by a heartbeat and can expire. Each acquisition also receives a fencing token: an increasing ownership number that an external target can use to reject a late write from an older owner. Expiration decides when another run may take over; fencing protects against the displaced run waking up and writing afterward.

A post-condition is a check the runner performs by observing an artifact or target, such as reading a file back or confirming a count changed. The agent’s ok: true is necessary but not sufficient. This proves a mechanical fact, not whether the analysis itself was wise.

I retained Anchor’s step and time budgets and added a circuit breaker that blocks new attempts after repeated failures until it is reset. Ratchet also provides a staleness check meant to run in a separate watchdog process. Code inside the runner can explain why a started run stopped; it cannot detect that the scheduler never started one.

Restarting after an external write

The failure window I designed around is an external action that lands just before the process dies:

write intent with stable idempotency key
  → invoke the step
  → external effect lands
  → process dies before returning a result

next run finds the orphan intent
  → read the external system
  → PRESENT: verify the effect and do not invoke again
  → ABSENT: retry only with the same logical key
  → UNKNOWN: retain the intent and stop for a human

The intent is a durable statement that an action may be in flight. Its idempotency key is a stable name for one logical action across retries, allowing a cooperating target to deduplicate the same publication. Reconciliation is the read-back that classifies the uncertain effect as present, absent or unknown. Keeping unknown separate from absent prevents uncertainty from becoming permission to repeat a write.

I do not claim exactly-once behavior across two systems. Ratchet preserves the ambiguity, gathers evidence from the target, and refuses a blind retry.

A key and a fence protect different failures

This shortened example comes from the repository’s executable crash-after-effect scenario:

from ratchet_runtime import EffectState, Reconciliation, Step

def effect_key(_ctx):
    return "publish:quarterly-report:2026-q2"

def publish(ctx):
    target.put(
        key=ctx.idempotency_key,
        fence=ctx.tenure_token,
        value=report,
    )
    return {"ok": True}

def reconcile(_ctx, intent):
    record = target.lookup(intent["idempotency_key"])
    if record is None:
        return Reconciliation(EffectState.ABSENT)
    if record.value == report:
        return Reconciliation(EffectState.PRESENT)
    return Reconciliation(EffectState.UNKNOWN)

step = Step(
    "publish",
    invoke=publish,
    side_effecting=True,
    idempotency_key=effect_key,
    reconcile=reconcile,
    postcondition=lambda ctx, _result: target.matches(
        key=ctx.idempotency_key, value=report
    ),
)

The idempotency key identifies the same publication across a retry. The fencing token identifies the current ownership generation. PRESENT skips invocation but still runs the normal post-condition. ABSENT permits a retry only if the newly evaluated key matches the orphaned one. UNKNOWN keeps the intent and stops. A target that ignores the key and token cannot inherit either protection from Ratchet’s local state.

I keep resolved intents until the whole workflow’s completion record is durable. Clearing an intent immediately after its step would lose the recovery evidence if a later step failed. If the process dies after completion but before cleanup, the next run can recognize the intent’s run ID as residue and remove it safely.

Why the local lock needs a coordinator

The public runtime targets a local POSIX filesystem on Linux or macOS. POSIX flock is an operating-system file lock; I use it around the short critical section that inspects ownership, allocates the next fencing token and publishes state.

I got the earlier lock design wrong twice. An atomic file replacement does not compare the value it replaces, so two simultaneous stale-lock recoveries could allocate the same fencing token before either published ownership. Exclusive creation with O_EXCL—an operating-system flag that lets only one process create a path—also had a smaller trap: creating the file and then writing the record briefly exposed an empty lock file.

The coordinator serializes ownership decisions, while a background heartbeat keeps tenure alive during a long-running agent call. Every runner-state mutation rechecks ownership under the same guard. A run that has lost tenure returns an in-memory outcome and writes no final diagnosis, because that diagnosis would itself violate the ownership rule.

This is a local coordination design, not a distributed lock. If I moved it across hosts, I would need a service that compares and replaces ownership atomically in one globally ordered operation while retaining the same fencing and effect-recovery contract.

What the conformance checks establish

The current public alpha passes 63 checks that invoke no language model. They race fresh acquisitions and stale recoveries across real processes, keep a heartbeat alive during a long step, interrupt a run after an external effect, and verify that missing post-conditions, empty keys, changed keys, ambiguous read-back, malformed results and unsafe pauses fail closed.

I treat the number as less important than the rule behind it: every claimed guarantee needs a mechanical conformance test that attacks its mechanism without relying on an agent to cooperate.

That rule came from green suites that were still wrong. Four tests once sat after the suite’s entry point and never registered. Later, 39 passing checks missed optional post-conditions, state writes after tenure loss, and duplicate token allocation during simultaneous stale recovery. The adversarial reviews now live beside the contract so those corrections remain part of the evidence rather than disappearing into commit history.

Current scope and limits

I currently publish 0.5.0a2, roughly 900 lines of standard-library Python, as an educational reference implementation and local experimental alpha. I have not established:

• safe locking on NFS, SMB or another network filesystem;
• durability during a real machine or power failure;
• idempotency against a live external API;
• semantic quality of an agent’s judgment;
• forced termination of an in-process step when its time budget expires;
• isolation from agent code running as the same operating-system user; or
• where the separate staleness watchdog is scheduled.

File permissions and hiding the state store from callback context do not isolate untrusted code running under the same user. That requires a separate process identity or a mediated API. The completion marker, source positions and outcome share one atomic file, but the circuit breaker and per-run history remain separate durable writes rather than one cross-file transaction.

Reading the repository

The public repository is intentionally separate from my faster-moving working copy. The useful reading order is:

1. The exact six guarantees in the contract.
2. The state boundary and effect lifecycle in the architecture.
3. The local-coordinator trade-off in ADR-001.
4. The first design review and release-candidate review.
5. The executable recovery example and conformance suite.

I keep Ratchet’s boundary deliberately narrow: the agent may decide how to perform a declared step, but not whether it ran, whether an uncertain external action is safe to repeat, or whether the run deserves a completion marker. It is still software for learning and experimentation, not production use.

Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.