Pi's Durable AgentHarness: An Agent Loop That Survives kill -9

Most agent runtimes model a run as a while loop: send the context, take the response, execute the tools, push onto an array, repeat. The array lives in memory. Kill the process between the tool executing and the push, and the tool ran but nothing in the system knows it did.

Pi is the agent harness from earendil-works — pi-agent-core underneath, a self-extensible coding agent on top. Inside packages/agent/docs/ there is a document called Durable AgentHarness design, and it is the most database-shaped agent design I have read: roughly 4,600 lines containing a record catalog, a validity section written as a list of rejection conditions, and tables mapping every crash prefix to exactly one recovery action.

I read it end to end as a learning exercise. This is the summary I wish I had started with — what it actually claims, which handful of ideas do the real work, and where the cost lands.

TL;DR

• One sentence carries the whole design. Before an effect, write an intent record naming what will happen and every id the settlement will use. After the effect, append the complete response, then its usage. Everything else is that rule applied to each kind of effect.
• Ids are allocated before the effect, not after. A provisioned id turns recovery from inference into a lookup: recovery already knows the id of the entry that should exist, so it can just ask whether it does.
• Conversation and orchestration are separate data. The tree holds messages only; lane records hold execution only. Delete every record and a complete, valid conversation remains.
• Append-only context is a cost invariant, not a style rule. A write inserted before the previous request's tail invalidates the provider's KV cache from that point on, so mid-turn writes defer to a checkpoint instead of landing where they happened.
• Abort is a record, not a boolean — and the sharp consequence is that a provider response with stop reason aborted and no abort record is not an abort at all. It is an interruption, and it retries.
• Determinism is a test surface. drive: "manual" parks the harness before every effect, so crash sites are enumerated mechanically from the effect boundary instead of being hand-picked by whoever wrote the test.
• The cost is real. Around 4,600 lines of specification for one component, and a work plan in section 20 with a git-based reservation protocol, because nobody is landing this in a single pass.

The loop that cannot be resumed

Start with what actually goes wrong. An agent turn contains at least three effects that reach outside the process: a provider request that costs money, tool executions that touch the filesystem or the network, and durable writes. A crash can land between any two of them, and each gap leaves a different mess.

The nastiest is the provider request. If the process dies mid-stream, you do not know whether the request was billed, whether it was served, or whether a partial answer exists somewhere. The design's response to this is the honest one: it is listed as a non-goal.

Partial streams are never persisted or resumed.

That refusal is what makes the rest tractable. Once you accept that one specific step is unrecoverable, you can arrange every other step so that recovery is a lookup rather than a guess. The whole document is that arrangement.

The two goals underneath it are stated plainly. Durable runs: an accepted prompt is a durable operation, and after a crash a new process reconstructs it from records and resumes from the last durable boundary. No partial outcomes: a crash inside any operation leaves one of two states — it has not happened, or recovery can complete it. Nothing in between is observable.

A session is four things, not one

The first clarifying move is refusing to model a session as one log.

The tree is the conversation and nothing else: entries chained by parentId, only ever appended, never edited. It belongs to no lane. The lanes own everything active — a leaf, a configuration, a queue, an operation. Lane records are the orchestration ledger, one flat chronological sequence per lane. Global facts are a latest-write-wins namespace for the session name, entry labels, and application-defined keys.

The invariant that falls out of this split is the one I would tattoo on a whiteboard:

Configuration and operation records never enter the tree. Deleting every operation log leaves a complete, valid conversation.

That is what makes forks, exports and backwards compatibility cheap. A fork copies conversation entries and no orchestration records, so the copy is idle by construction with a zero-cost ledger. Old v3 session files, which have no records at all, open as one normalized idle lane. Neither case needs a migration path, because the conversation was never entangled with the machinery in the first place.

The four parts share one monotonic seq, which is the only coupling between them: it orders global-fact history and lets a lane's records point at tree positions.

Lanes: parallelism without a second session

A lane is a named position in the tree with at most one operation on it. The document's own analogy is a git branch in its own worktree — new work advances it, navigation moves it to any existing entry without rewriting history.

The motivating case is chat. A Slack channel becomes one session with one lane per thread; interactive pi uses a single hidden lane. The alternative designs are both worse: one session per thread throws away the shared history that makes a channel useful, and one lane behind a mutex throws away the parallelism.

What makes lanes work is what they explicitly do not share. A lane owns its leaf, its operation log, its queues, and its total configuration. Creating a lane copies nothing from its anchor — not history, not configuration — and every lane starts from the same immutable seed captured from the harness options. Two lanes sitting at the same leaf simply diverge on their next append; the tree handles that, and no coordination exists between them.

Configuration deserves a note because the rule is unusually strict. A lane's configuration is one value holding the model reference, the thinking level and the active tool names, and a lane_config record always replaces the whole value. Never a patch. A setter commits immediately, even mid-operation, but a generation step snapshots the configuration when its step_started commits and every retry of that step uses the snapshot. So changing the model while a request is in flight is well-defined: the in-flight step keeps the old model through all its retries, and the next step picks up the new one.

The durability rule

Here is the sentence the whole design reduces to:

Before an effect: write an intent record that names what will happen and every durable id settlement will use. After an assistant/fetch effect: append the complete response entry, then its preplanned usage record.

And here is one assistant step written out, with what recovery does in each gap between two writes:

The load-bearing idea is provisioned ids. Before the provider request happens, step_attempt records the id that the response entry will have and the id that the usage record will have. Neither object exists yet. This is what converts recovery from inference into a query — restore does not have to reconstruct what probably happened, it reads the intent, collects the ids it names, and asks storage which of them exist.

It also gives corruption a crisp definition:

A provisioned id that exists with different content is corruption.

The doc's own storage-level traces use a compact legend, and a whole run with one tool call reads like this — R is a record, E is a tree entry, H is an awaited hook:

    prompt("fix the bug")
H   before_run                        may inject entries, override system prompt
R   operation_started                 kind run; initial messages with their ids
E   user message                      the provisioned id from the intent
R   step_started                      assistant; config, policy, trigger = U
R   step_attempt                      attempt 1; response and usage ids reserved
E   assistant message [tool call]     complete settled response, any stop reason
R   usage                             preplanned id; before classification
R   tool_batch_started                c1 source index and provisioned result id
H   before_tool                       may change args or block
R   tool_started                      c1 effective args and replay declaration
    execute tool                      c1's individually gated phase-two effect
H   after_tool                        may patch result, usage, and terminate
R   usage                             c1 tool usage when reported; before result
E   tool result                       c1's planned result id; persists terminate
R   step_started                      next assistant step; new id and trigger
R   step_attempt                      attempt 1; fresh response and usage ids
E   assistant message "done"
R   usage
H   before_run_end                    nothing pending, returns nothing
R   operation_finished                completed

Note where the tool batch is planned: tool_batch_started assigns a result entry id to every call in the response, before tool lookup, before argument validation, before the before_tool hook, and before any execution. Calls that later turn out to be blocked, invalid, interrupted or aborted still get their planned id. Recovery therefore never has to decide whether a missing tool result was ever promised — the promise is durable, and reconciliation just fills it.

Every crash prefix has one reading

The claim that makes this more than a logging convention is that the write order admits no ambiguous state. Classification of an assistant response begins only after both durable objects exist — the complete message entry, then its preplanned usage record — and is a pure function of them. The doc enumerates the prefixes:

That fifth row is the tell. A design that merely writes things down as it goes cannot say usage without a response is impossible — it can only say it would be surprising. Here the ordering is a contract, so its violation is diagnosable.

The "linked transition" row is the other subtle one. There is no general outcome record saying what the harness decided about a response. Instead, the decision is inferred from what already exists downstream: a newer attempt of the same step means retry, an overflow compaction naming this exact response means overflow recovery, a durable tool-batch plan means the calls were accepted. Recovery re-derives the decision from its consequence rather than storing it, which removes an entire class of write-skew between decision and effect.

Append-only context is a cost invariant

This one is my favourite because it is the place where a distributed-systems concern and a billing concern turn out to be the same concern.

Across the requests of a lane, provider context only grows at the tail. An insertion before the previous request's tail invalidates the provider's KV cache from that point on and multiplies token cost.

So when the application appends a message while a request is in flight, the harness does not append it where it happened. It records a write_deferred intent with the full payload, and applies it at the next checkpoint, at the tail:

R   step_started                      assistant; trigger U
R   step_attempt                      response A and usage id; request in flight
    session.appendMessage(M)          caller resolves here
R   write_deferred                    full payload, provisioned id
E   assistant message A               provider cached [.., U, A]
R   usage                             before classification
E   message M                         checkpoint applies the write; tail append

Appending M directly would produce [.., U, M, A] — a perfectly valid message sequence that invalidates the cache from M onward and produces a transcript claiming that A saw M when it did not. One rule fixes a correctness bug and a cost bug simultaneously.

The corollary is that persistence and provider-context projection are two different things. Assistant responses with stop reason error, aborted or deferred are all durably appended and all project to nothing when the next request is built. Compaction is described as the one deliberate cache invalidation, traded knowingly for a smaller context. And Tier B of the test strategy turns this into an executable assertion: within a run, every request's message list must extend the previous request's as an exact prefix, except across a compaction entry.

One compaction per user input

Context overflow is where a naive design loops forever: the request does not fit, so compact and retry, and if it still does not fit, compact and retry again.

The fix is a single field. Every assistant generation step records a triggerMessageId — the id of the newest consumed user-context message that caused it. An overflow compaction is allowed only if no earlier compaction in this run has the same trigger:

consume user message U1
start assistant step A1, triggerMessageId = U1
persist recoverable-overflow response R1
compact once, linked to R1 and U1
start assistant step A2, triggerMessageId = U1
persist recoverable-overflow response R2 → fail; no second compaction
consume steering message U2
start assistant step A3, triggerMessageId = U2 → one new compaction allowed

New human input buys exactly one new attempt at making the context fit. That is a loop bound expressed as a data dependency rather than a counter, which means it survives restarts for free — the trigger is on the durable record.

The classification underneath it is refreshingly unglamorous. Three durable signals mean overflow: an explicit provider context-limit error whose message matches known patterns (prompt is too long, exceeds the context window, DashScope/Qwen's Range of input length should be), a successful response whose reported input plus cache-read tokens exceed the attempt's captured context window, or a length stop that ended below the request's intended output limit:

function isRecoverableLength(
  message: AssistantMessage,
  intendedOutputLimit: number,
): boolean {
  return message.stopReason === "length"
    && intendedOutputLimit > 0
    && message.usage.output < intendedOutputLimit;
}

The reason the intended limit is captured rather than the sent one is a nice piece of provider archaeology: some providers reject explicit caps outright — OpenAI Codex returns HTTP 400 for max_output_tokens — and pi clamps others to the remaining context. So sixteen reasoning tokens against a 128k intent is recoverable overflow, while a fully consumed explicit 1,024-token cap is a genuine output limit and stays in context.

Abort is a record, not a flag

Cancellation in an in-memory loop is a boolean somebody checks. Here it is a durable abort_requested record, written before anything is signalled, and that record — not the flag, not the stop reason — is the authority.

The consequence I did not expect is the inverse rule. A transport timeout, a harness close, or a provider-side cancellation can all produce a settled response with stop reason aborted. If no abort marker precedes it, that response is not an abort:

Stop reason alone is not abort authority: unmarked aborted follows interruption retry/failure and never aborts the operation.

So the same stop reason means two different things depending on a record written elsewhere, and the design makes you look at the record. An unmarked aborted retries under the captured policy and, at the cap, finishes the run failed — not aborted. Conflating those two would silently turn a flaky network into a user cancellation in every log and metric downstream.

Abort also refuses to fabricate closure. There is no synthetic assistant message summarising the cancellation, and the harness never starts a request just to manufacture one. What reconciliation does is bounded and mechanical: complete accepted initial messages, complete promised tool results (started calls that never returned get interrupted, calls that never started get aborted), apply accepted deferred writes, then write operation_finished with outcome aborted. That record is the only universal terminal marker — no tree entry serves that role.

The mutation line

Every remaining race is a check-then-act separated by an await, and the fix is eight lines:

let tail: Promise<unknown> = Promise.resolve();

function mutateLane<T>(job: () => Promise<T>): Promise<T> {
  const result = tail.then(job);
  tail = result.then(() => undefined, () => undefined);
  return result;
}

One FIFO per lane. A job validates live state, performs at most one storage append, installs the result, and publishes events — all without yielding. Provider requests, tool executions and retry backoff run between jobs, never inside one, so every commit revalidates against current state.

What this buys is that concurrent operations have exactly two possible histories and no third. The doc lists both for each of twelve races; steering against run completion is the clearest:

steer first                         finish first
R   queue_enqueued                  R   operation_finished
    tryFinishRun → continue             steer() → NoActiveRun
E   user message
... run continues
R   operation_finished

Only one race in the catalog is called irreducible — abort against an in-flight provider or tool effect — because an external effect can occur without returning a result. And the design's answer there is to not treat it as special: the intent record plus the replay policy handle it exactly like a crash.

Two serialization layers are needed and the doc is explicit that neither substitutes for the other. Storage linearizes commits across lanes with one monotonic seq; the mutation line linearizes decisions within a lane. Only the second one can close a check-then-act.

Determinism as a test surface

The part I would steal wholesale for any stateful system is the testing design, because it is a consequence of the architecture rather than an addition to it.

Every effect — durable write, provider request, tool execution, hook, timer — crosses one injected Effects boundary. In drive: "manual" that boundary becomes a gate: the harness parks before each effect and exposes a JSON-safe description of what it is about to do. A test drives it call by call with peekAction() and executeAction(), and the construction rule is enforced by a test of its own — while parked, zero storage writes and zero provider or tool calls happen.

The gate is not a mock. Production and tests run the same procedures; the drive mode only controls the boundary. Which means crash sites stop being hand-picked:

Crash sites are derived mechanically, not hand-picked: drive each section 6 trace in manual mode, capture the backend before and after every executeAction() — atomic storage append, hook, provider/fetch, individual tool, or timer — […] then reopen every boundary case and resume().

And then it recurses. Whenever recovery itself commits an entry, record, lane move or fact, the test closes immediately, reopens that new prefix, and continues — so every recovery write is also a crash boundary. Running recovery twice from the same starting prefix is explicitly called insufficient. Add a new effect to a procedure and it gets crash coverage automatically, which is the property that makes this survive maintenance.

The three tiers divide the claims cleanly. Tier A prefills a crash state and asserts what resume() produces. Tier B runs the public harness against an instrumented session and asserts the exact write order against the documented traces — catching, in the doc's own words, an effect starting before its intent record, or classification starting before usage is durable. Tier C drives both orders of every race-catalog row.

What it costs

I would not trust this write-up if it stopped at the good parts.

It is very large. Roughly 4,600 lines of specification for one component, with a validity section that reads as about thirty rejection conditions in a single sentence-per-bullet block. Section 20 organises the implementation into tracks with a git-based claiming protocol — you add Reserved: <package-id> by @<username> above the entry, land that change alone, and only then start — plus the instruction that if the design does not hold you stop and consult the maintainer on Discord before changing it. That is a sane process for a spec this size, and it is also an admission of how much coordination the size demands.

The complexity is not evenly earned. Lanes, provisioned ids and the mutation line pay for themselves immediately. The deferred-request machinery — a source lineage of deferred responses, complete-handle equality, a per-source attempt cap distinct from the generation cap, one fetchDeferred(wait: 0) per resume() — is a lot of specification for batch-style provider APIs that many applications will never touch. It is correct, and it is optional weight.

Some of the hardest problems are scoped out rather than solved, which I think is the right call but should be said plainly. Multiple writers per session: out of scope, enforced by the serving layer and a fenced SQLite writer lease. Replication: out of scope. Exactly-once hook side effects: explicitly a non-goal, with a replay table telling you exactly which hooks can re-run after a crash so your handler can be idempotent. And durability is process-crash level — in the JSONL backend, a resolved append call with no fsync promise. This design survives kill -9, not a power cut, and it says so.

What actually scales down

Most of this is unnecessary if your agent runs for ninety seconds inside one process. But four ideas cost almost nothing and are worth taking even at that scale:

Write intent before the effect, and put the ids in the intent. This is the single highest-leverage line in the document. It is one extra row before every side effect, and it converts "what state am I in?" from an inference problem into a lookup.

Keep orchestration out of the conversation. Retry counts, step ids and tool plans do not belong in the message list. Once they are separate, forking, exporting, replaying and reading old formats all become easy for the same reason.

Treat persistence order and context order as different problems. What you store and what you send are not the same list, and the difference is worth money: never insert before the tail of the previous request.

Make the effect boundary injectable from day one. Not for mocking — for stepping. If every effect crosses one seam, you can enumerate your own crash sites instead of imagining them, and that is the difference between a recovery path you believe in and one you have actually run.

The last one is really the meta-lesson of the whole document. The reason it can make a claim as strong as every crash prefix has exactly one reading is that it first made the set of crash prefixes finite and enumerable. Durability came second.