Pi's Durable AgentHarness: An Agent Loop That Survives kill -9

Most agent runtimes model a run as a while loop: send the context, take the response, execute the tools, push onto an array, repeat. The array lives in memory. Kill the process between the tool executing and the push, and the tool ran but nothing in the system knows it did.

Pi is the agent harness from earendil-works — pi-agent-core underneath, a self-extensible coding agent on top. Inside packages/agent/docs/ there is a document called AgentHarness — implementation specification, and it is the most database-shaped agent runtime I have read: roughly 2,900 lines containing a three-store storage model, a total operation-state machine, and a recovery story that is a point lookup rather than a journal replay.

I first wrote this up against an earlier draft of that spec. The draft was a record catalog: every crash prefix of an append-only log had exactly one reading. A day later the document was consolidated into the current text, and the load-bearing idea changed. Recovery no longer classifies a prefix. It reads one register. This is the summary of the spec that is on main now.

TL;DR

• Everything durable is one of three things. Write-once entries (the conversation), overwriteable registers (current mutable state), and an append-only usage ledger. There is no fourth place a payload can hide.
• op.state is the program counter. After every step the harness overwrites that register with the complete current state of the operation. Recovery is five point lookups. It does not replay a journal, fold history, or infer position from what is missing.
• An effect is two commits around one uncertain call. Commit the intent — mint the ids settlement will use, write effect_pending. Do the request or the tool. Commit the settlement — the complete entry, its usage row, and the next total state — in one transaction.
• Atomic transactions have no internal prefix. The only uncertain interval in the system is intent durable and settlement absent. Three policies cover it, and they are read off the captured state, not reconstructed.
• Abort is control, not a phase. abort() commits cancel_requested before it pulls the signal, so a settled aborted response always has cancellation already durable. A timeout that happens to say aborted is not an abort; the provider contract forbids it.
• Conversation and orchestration stay apart. Configuration and op.* registers never enter the tree. Delete every operation register and a complete, valid conversation remains.
• The cost is real. Roughly 2,900 lines of specification for one component, a nine-part machine, and a test plan that treats every race as two histories.

The loop that cannot be resumed

Start with what actually goes wrong. An agent turn contains at least three effects that reach outside the process: a provider request that costs money, tool executions that touch the filesystem or the network, and durable writes. A crash can land between any two of them, and each gap leaves a different mess.

The nastiest is the provider request. If the process dies mid-stream, you do not know whether the request was billed, whether it was served, or whether a partial answer exists somewhere. The spec's response to this is the honest one: it is listed as a non-goal.

Partial streams are process-local, never persisted. A settled response is persisted completely before anything classifies it.

That refusal is what makes the rest tractable. Once you accept that one specific step is unrecoverable, you can arrange every other step so that recovery is a lookup rather than a guess. The whole document is that arrangement.

The two claims underneath it are stated plainly. Durable runs: an accepted prompt is a durable operation, and after a crash a new process reconstructs it from registers and resumes from the last durable boundary. No partial outcomes: a crash inside any operation leaves one of two states — the previous state, or a named effect_pending whose ids are already reserved. Nothing in between is observable, because a transaction is all-or-none.

Three stores, not a journal

The first clarifying move is refusing to model a session as one log.

A session still has four parts — an entry tree, facts, lanes, and a usage ledger — but those are a view. Underneath them, Storage exposes exactly three durable forms:

Entries are the conversation and nothing else: messages, compaction summaries, branch summaries, chained by parentId. Written once. Never edited. Never deleted. They belong to no lane.

Registers hold current mutable state. A lane is three of them (leaf, config, state). An open operation is two more (op.meta, op.state). Facts live here too — session name, entry labels, application keys — latest write wins, and a delete removes the key. There are no tombstones and no write history. Overwriting op.state discards the previous value.

The usage ledger is append-only cost: one row per settled attempt. It never enters the tree and it is never folded back into recovery.

The invariant that falls out of this split is the one I would tattoo on a whiteboard:

Configuration and orchestration never enter the tree. Deleting every op.* and pending.entry register leaves a complete, valid conversation and ledger.

That is what makes forks, exports and v3 compatibility cheap. A fork copies conversation entries and no operation registers, so the copy is idle by construction. Old v3 session files, which have no registers at all, open as one normalized idle main lane. Neither case needs a migration path through the operation machine, because the conversation was never entangled with it.

The three stores share one monotonic seq. A transaction is a set of entry inserts, usage inserts, and register writes, committed all-or-none. That is the only write primitive. There is no crash state inside a transaction — including on the JSONL backend, where a torn final line is discarded whole.

Lanes: parallelism without a second session

A lane is a named cursor in the tree with at most one operation on it. The document's own analogy is still the right one: a git branch in its own worktree — new work advances it, navigation moves it to any existing entry without rewriting history. Every session has main.

The motivating case is chat. A Slack channel becomes one session with one lane per thread; interactive pi uses a single hidden lane. The alternative designs are both worse: one session per thread throws away the shared history that makes a channel useful, and one lane behind a mutex throws away the parallelism.

What makes lanes work is what they explicitly do not share. A lane owns its leaf, its configuration, its queues, and at most one operation. Creating a lane copies nothing from its anchor — not history, not configuration — and every lane starts from the same immutable seed captured from the harness options. Two lanes sitting at the same leaf simply diverge on their next append.

Configuration is unusually strict. LaneConfiguration is one total value: the model identity, the thinking level, the active tool names. A setter overwrites the whole register. Never a patch. Never a tree entry. A generation step snapshots that value into op.state when it leaves ready, and every retry of that step uses the snapshot. Changing the model while a request is in flight is well-defined: the in-flight attempt keeps the old model through all its retries, and the next step picks up the new one.

The durability rule

Here is the sentence the whole spec reduces to:

Commit: "about to do X; its output will use ids R and U." Do X. Commit: the complete output, its usage, and the next total state.

And here is one assistant step written out, with what recovery does in each gap:

The load-bearing idea is still reserved ids, but they no longer live in a journal row. Before the provider request happens, the intent commit writes assistant{effect_pending} and stores the response-entry id and the usage id as strings inside op.state. Neither object exists yet. Restore does not have to reconstruct what probably happened. It reads the register, collects the ids it names, and asks storage which of them exist.

It also gives corruption a crisp definition:

A reserved id may exist only with the content its intent named.

And it kills a whole class of half-written states that the earlier draft had to enumerate. Classification of an assistant response is a pure function computed in memory before the settlement transaction. Then response, usage and the next state land together:

There is never a durable "response without usage" or "response and usage without a decision." All three land together or none do.

That is the change that let the spec drop the prefix table. A design that writes the response first and the usage later can say usage without a response is surprising. A design that writes them in one transaction can say it is impossible.

The next state is inferred the same way the old draft inferred a "linked transition" — from what the settlement actually committed. Tools means the settlement minted a result-entry id per call. Retry means it wrote retry_wait with nextAttempt. Overflow means it normalized the response to error and entered compaction. Recovery does not re-derive a missing decision; the decision is the next total state.

The one uncertain window

Atomic transactions have no internal prefix, so for any repeat-sensitive effect there are only a handful of durable positions. The spec lists them. The one that does the real work is this:

The one uncertain interval in the entire system is: intent durable, settlement absent.

Restore finds effect_pending and no live in-process key (the running map died with the process). Three policies cover it, read off the captured state:

The mid-tool crash in section 0.5 is the one I would steal as a teaching example. The model returns two tool calls. The harness commits the batch plan, then commits call 0 is about to execute, with these exact arguments, and it declares itself unsafe to replay. The tool starts deleting files. The process is killed.

On restart the harness reads one register and finds calls[0].status = "effect_pending", replay = "never". It does not re-run the deletion. It appends a synthetic error result under the result id that was reserved before the effect started, marks the call complete, and continues to call 1. The conversation stays coherent — every tool call has a result — and nothing ran twice.

Had the tool declared replay: "safe", the harness would have re-executed it with the persisted arguments instead. Exactly-once hook side effects remain a non-goal: a hook that has not been consumed by a transaction may rerun, and your handler has to be idempotent.

Append-only context is a cost invariant

This one is my favourite because it is the place where a distributed-systems concern and a billing concern turn out to be the same concern.

Across the requests of a lane, provider context only grows at the tail. An insertion before the previous request's tail invalidates the provider's KV cache from that point on and multiplies token cost.

So when the application appends a message while a request is in flight, the harness does not append it where it happened. It writes a pending.entry register with the full payload, and the checkpoint procedure applies it at the tail:

S   assistant ready                   trigger U; request about to start
S   effect_pending                    R and U reserved; request in flight
    session.appendMessage(M)          caller resolves here
    pending.entry/{id} = M            full payload, reserved id
TX  settlement                        provider cached [.., U, A]
    checkpoint applies the write      M is appended at the tail

Appending M directly would produce [.., U, M, A] — a perfectly valid message sequence that invalidates the cache from M onward and produces a transcript claiming that A saw M when it did not. One rule fixes a correctness bug and a cost bug simultaneously.

The corollary is that persistence and provider-context projection are two different things. Assistant responses normalized to error (overflow) or aborted are all durably appended and all project to nothing when the next request is built. Compaction is described as the one deliberate cache invalidation, traded knowingly for a smaller context.

One overflow compaction per user input

Context overflow is where a naive design loops forever: the request does not fit, so compact and retry, and if it still does not fit, compact and retry again.

The fix is a single flag. Every assistant generation step carries a triggerEntryId — the id of the newest consumed user-context message that caused it — and an overflowRecoveryUsed bit. An overflow compaction is allowed only if that bit is still false for this trigger:

consume user message U1
start assistant step, triggerEntryId = U1
persist overflow, normalize the response to error
compact once, overflowRecoveryUsed = true
start assistant step, same trigger
persist overflow again → failure_drain; no second compaction
consume steering message U2
start assistant step, triggerEntryId = U2 → one new compaction allowed

New human input buys exactly one new attempt at making the context fit. That is a loop bound expressed as a data dependency rather than a counter, which means it survives restarts for free — the flag is on the durable state.

The classification underneath it is refreshingly unglamorous. Three signals mean overflow, in decreasing reliability: an adapter that can compute usage.input + usage.cacheRead > contextWindow, an error whose message matches known context-limit patterns, or a length stop that ended below the attempt's captured intendedOutputLimit. The intended limit is captured rather than the sent one because some providers reject explicit caps outright, and the harness clamps others to the remaining context. So sixteen reasoning tokens against a 128k intent is recoverable overflow, while a fully consumed explicit 1,024-token cap is a genuine output limit and stays in context.

Because the overflow response is committed as error, the ordinary projection rule drops it from the next request automatically. No dedicated omission list. The response stays in the tree, because a provider request happened and was billed.

Abort is control, not a phase

Cancellation in an in-memory loop is a boolean somebody checks. Here it is control = cancel_requested, written before anything is signalled, and that register — not the flag, not the stop reason — is the authority.

The consequence I did not expect is how unambiguous aborted becomes. The harness owns the abort signal exclusively. Providers must set stopReason: "aborted" if and only if that signal was pulled. Since abort() commits control first, a settled aborted response always has cancellation already durable.

An aborted response with control.status === "running" is unreachable; if one exists, the session is corrupt.

Timeouts, transport failures, malformed streams and provider-side refusals all settle as error and take the ordinary retry path. Conflating those with abort would silently turn a flaky network into a user cancellation in every log and metric downstream. The earlier draft had to say "unmarked aborted is an interruption." The current spec makes the unmarked case unreachable.

Abort also refuses to fabricate closure. There is no synthetic assistant message summarising the cancellation, and the harness never starts a request just to manufacture one. What reconciliation does is bounded and mechanical: settle effects that were already intended, complete promised tool results (started unsafe calls get interrupted, planned calls get aborted), apply accepted deferred writes, then commit the terminal transaction. That transaction is the only universal terminal marker — the operation ceases to exist. Its registers are deleted. The outcome lives in lane.lastResult.

Close is not abort. Close writes nothing. It pulls the same signal so in-flight requests stop, then seals admission so a locally-aborted response cannot commit with running control. Durable state stops at effect_pending, exactly as after kill -9. Reopening applies the uncertain-window policy.

The mutation line

Every remaining race is a check-then-act separated by an await, and the fix is still a FIFO per lane: validate, at most one atomic commit, publish the in-memory result — all before the next mutation starts. Provider requests, tool executions, hooks and retry backoff run between jobs, never inside one.

What this buys is that concurrent operations have exactly two possible histories and no third. The spec lists both for each of twelve races; abort against response settlement is the clearest:

abort first                         settlement first
S   control = cancel_requested      TX  response + usage + next state
    settlement normalizes aborted       abort() → drained, same payloads

Two serialization layers are needed and the spec is explicit that neither substitutes for the other. Storage linearizes commits across lanes with one monotonic seq; the mutation line linearizes decisions within a lane. Only the second one can close a check-then-act.

Determinism as a test surface

The part I would steal wholesale for any stateful system is the testing design, because it is a consequence of the architecture rather than an addition to it.

Every effect — durable write, provider request, tool execution, hook, timer — crosses one injected Effects boundary. In drive: "manual" that boundary becomes a gate: the harness parks before each effect and exposes a JSON-safe description of what it is about to do. A test drives it call by call, and the construction rule is enforced by a test of its own — while parked, zero storage writes and zero provider or tool calls happen.

The gate is not a mock. Production and tests run the same procedures; the drive mode only controls the boundary. Lane-surface calls stay ungated, so a test can drive both orders of any race.

Crash sites stop being hand-picked. Tier A constructs every state in Part 3 durably, closes, reopens, and asserts the next action — including assistant intent with no settlement, below and at the retry cap, every tool state, every overflow crash position, abort at every position. For each recovery prefix: close, reopen, resume, and compare against uninterrupted recovery. Running recovery twice from the same starting prefix is explicitly called insufficient.

Tier B is the tell that the journal is gone. There is no durable log to compare against. The oracle is an instrumented decorator around Storage.commit() that records every transaction's writes in order. That is how the suite catches an effect starting before its intent commit, or classification starting before usage is durable.

Tier C drives both orders of every race-catalog row. Drive equivalence is a separate assertion: the same scenario in automatic and manual drive must produce byte-identical durable state.

What it costs

I would not trust this write-up if it stopped at the good parts.

It is very large. Roughly 2,900 lines of implementation specification for one component, nine parts plus three appendices, and a validity section restated as twenty-one numbered invariants. The earlier draft organised the work with a git-based claiming protocol. The current text is the thing you implement against, not the thing you reserve a section of.

The complexity is not evenly earned. Three stores, reserved ids and the mutation line pay for themselves immediately. The deferred-request machinery — a source lineage of deferred responses, complete-handle equality, a per-source poll that reserves fresh ids, one cancel path that is the only external action allowed under cancelled control — is a lot of specification for batch-style provider APIs that many applications will never touch. It is correct, and it is optional weight.

Some of the hardest problems are scoped out rather than solved, which I think is the right call but should be said plainly. Multiple writers per session: out of scope, enforced by the serving layer and a fenced SQLite writer lease. Replication: out of scope. Exactly-once hook side effects: explicitly a non-goal. Durable write history: registers hold only current values; order-of-write assertions in tests use the instrumented decorator, not a log. And durability is process-crash level — in the JSONL backend, a resolved commit() with no fsync promise. This design survives kill -9, not a power cut, and it says so.

One cost the consolidation added, rather than removed: JSONL now grows with write history even though the logical state does not. Every op.state overwrite appends a line. A 30-turn run that leaves one row in SQLite leaves ~10 dead op.state lines in the file, retired only when snapshot compaction rewrites it. Logical deletion is immediate. Physical deletion is deferred.

What actually scales down

Most of this is unnecessary if your agent runs for ninety seconds inside one process. But four ideas cost almost nothing and are worth taking even at that scale:

Write intent before the effect, and put the ids in the intent. This is still the single highest-leverage line in the document. It is one extra commit before every side effect, and it converts "what state am I in?" from an inference problem into a lookup.

Keep orchestration out of the conversation. Retry counts, step ids and tool plans do not belong in the message list. Once they are registers, forking, exporting, replaying and reading old formats all become easy for the same reason.

Treat persistence order and context order as different problems. What you store and what you send are not the same list, and the difference is worth money: never insert before the tail of the previous request.

Make the effect boundary injectable from day one. Not for mocking — for stepping. If every effect crosses one seam, you can enumerate your own crash sites instead of imagining them, and that is the difference between a recovery path you believe in and one you have actually run.

The last one is really the meta-lesson of the whole document. The earlier draft could claim every crash prefix has exactly one reading because it first made the set of prefixes finite. The current spec makes a stronger move: it made the set of crash positions smaller, by refusing to have a prefix inside a transaction at all. Durability came second. The store model came first.