sausage/docs/SAU-384-runbook.md

One-off handover runbook — rehearsed; live approval required

Authority: design / Rollout, SAU-356 gates. Results: initial, races/rollback, native package, external driver rework. This is a proposed live plan, not authorization. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.

0. Prepare everything while the old world is still available

Do not begin downtime with scripts, caller decisions or launch instructions still to be assembled.

Reviewed selection

• Bundle: /docs/symphony/artifacts/SAU-384/cutover-targets-1381.tar.gz, SHA-256 9295c5fddbf5d68cbfd525602ec67e7dd895bf6afb99324191a6c514282eac57. It includes final-main cred 0db40101, build 1636 / 274 tests, preserving approved SAU-418 instead of downgrading it. The earlier cutover-targets.tar.gz remains immutable prior evidence, not this final selection. Verify all 22 payloads against its embedded manifest.json. target-artifacts.tar.gz is older debug-host evidence, not this installation.
• Native host: /nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0, aarch64-linux Nix release package from unchanged host source e09c4f9f. The embedded native-host-manifest.json pins the NAR, binary and complete runtime closure. Import the reviewed NAR if needed; never overwrite immutable store files or substitute a diagnostic host. Import/deployment is an operator action, never a worker action.
• Old package for this observed baseline: /nix/store/mggvibk8lcgv2qw28jimrl60mi0qm41p-sausage-0.1.0. Re-identify it and the owner before go; observed PIDs are not constants.
• Actual owner: Herdr w7:p6, tab w7:t5, not a systemd Sausage unit. Its loop is /nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/. Read herdr pane process-info --pane w7:p6 once to resolve current shell/loop/host identities. Do not close Herdr or unrelated panes.
• SDK copy list is src/lib.rs only. Preserve world-owned Cargo manifests, relocated interface, imports and all other SDK bytes. Resolve a changed baseline before proceeding.
• Web is approved commit 3d76026749a5d21722033bbfe8743ac368946b0f (SAU-375), held by publish:hold. Do not release it before target session and web are active.

External driver and resolved operator manifest

Use external root /home/surma/sau-384-cutover, outside the world and stopped process group. Stage the extracted bundle under targets/. Reconstruct only cutover_once.py, freeze.py, and driver.py under tool/ from the source-as-log headings in /logs/box/2026-10-31/tc-asa3yr-exec-3977808892.stderr; check their hashes against evidence/rework-1381/tool-deletion.json. These are the executed/reviewed versions, not the earlier scripts from run 1373. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.

The tested interpreter is /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3, with inotify-simple 2.0.1. Resolve its closure availability before go. After go and before starting the driver, pin its package and the inspected Herdr package with host-visible GC roots for the operation (nix-store --realise <exact-package> --add-root <external-root>/gc-controller --indirect, and equivalently gc-herdr). These root writes/imports are included in the approved live plan, not preparation actions authorized to a worker. The driver itself registers and verifies old host, target host and loop roots at <root>/gc-{old,target,loop} before freeze. Use the host namespace, not a box talking to shared Nix state: /rework roots in a box are not host-visible. Do not run GC as a test. Keep old/target/loop roots and both selectors for recovery after tool deletion.

Start from evidence/rework-1381/main-refresh/live-manifest.template.json, a non-executable template, and resolve every placeholder before approval:

• native paths, Herdr binary/context, pane and current shell/loop/host PIDs;
• external root, world, bundle, embedded manifest SHA, artifact paths, SDK path, exact target boot list;
• complete old in-flight /proc/*/info.json and native-owner inventory, plus uncertain external effects; a finished-looking assistant tail is not settlement;
• a new continuation UUID, generated once, and the explicit new mediator verification instruction. Keep that UUID on all retries;
• typed executable-caller review and its frozen guards below.

Do not copy sandbox PIDs, gc_store, or test_gate_after_freeze into a live manifest. The template deliberately omits those test-only fields. Record/hash the fully resolved operator.json; driver checkpoints bind to its exact bytes. Resolve a schema/selection discrepancy while the old world is available, not during downtime.

Executable caller review and frozen guards

Use predecessor formats: session source f9122585 Paths/queue; cron ba75c102 and current cron README one-off procedure. Do not substring-rewrite prompts or diagnostics. Review current stores as in resume-1373.md, then encode the approved file hashes and deterministic SQL projections in caller_file_guards and caller_sql_guards:

• cron state/cron/jobs.sqlite: select id,data from jobs order by id;
• board state/board/board.sqlite: deterministic projections of hooks, invariant_target, subscriptions, and subscription_batches, using the actual current schema. Review hook/invariant templates. Historical hook-attempt snapshots remain diagnostics and must not replay;
• boot and other discovered executable configuration/manager registrations. Review typed room/workflow/orchestrator stores too; preserve their histories/settings and non-executable records. Do not mistake prompt text for an RPC template.

A SQL guard hashes sha256(json.dumps(rows,separators=(',',':')).encode()), where rows = connection.execute(sql).fetchall() in the approved ordered projection. JSON strings inside rows retain their exact bytes. The observed live baseline had six cron templates to convert, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, session.work, session.cancel, ambiguous interrupt:true, invalid text or unexplained notify destinations stop preflight for explicit resolution.

After freeze the driver backs up all bytes, checks these projections on disposable complete DB copies, and verifies file guards against the frozen backup before apply. It never recovers/opens immutable backup evidence in SQLite. A changed projection stops offline with evidence; there is no in-world human gate after stop-all. Include any newly discovered executable store in the approved manifest before arm.

Approval and external launch surface

Surma's go must explicitly cover this resolved manifest, the stop-all core/session/cred cut, the detached external controller launch/arm, pinned launcher installation and the post-start verification/release/deletion plan. This card's outside-world-controller scope requires a controller that survives the mediator; it does not authorize executing it now.

The proposed concrete execution surface is two finite foreground ssh.exec calls as surma on citadel.vpn.surma.technology, using the existing approved signing grant, with commands cd /home/surma/sau-384-cutover && <the start command below> and then the same prefix with <the arm command below>. Each command terminates; the reviewed native CLI, not a shell &, creates the detached OS child. Ask Surma's go to explicitly authorize this planned exception to the ssh-exec skill's usual prohibition on creating background work through exec strings. Until that explicit authorization, do not launch. Do not improvise a tunnel, SSH PTY, new worker session or an in-world controller. This is a fully prepared proposed live operation, not a claim that the current worker has permission or has tested a live SSH launch. The same finite foreground surface executes the final verify/delete command after target verification. After arm, loss of the old mediator/SSH RPC result is expected and does not terminate the already-detached controller; inspect the external checkpoint instead of retrying start.

1. Start, acknowledge, then arm the external controller

Close ingress and record accepted first-update cuts before arm. From the external root, on the approved host execution surface, mediator executes:

PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py start operator.json

start launches a new OS session with stdin closed and output in <root>/driver.log, records its PID/start identity, returns an acknowledgement, and leaves the controller blocked on an arm FIFO. Check that receipt and checkpoint once, then:

PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py arm operator.json

From here the external driver, not the old mediator, owns steps 2–4. The old default run and its tool calls are deliberately cut. They are not recovered from legacy history. The driver uses an external advisory lock to exclude another controller. Rehearsal killed both the invoking caller shim and, separately, the detached controller after freeze; checkpoint resume completed the chain.

2. Driver: freeze → full backup → plan/audit → offline apply

The driver verifies reviewed bundle/plugin/SDK bytes and native closure hashes, registers real GC roots, and creates hash-guarded old and target selector/owner artifacts before freeze. It then:

1. Pins boot ID, start ticks, executable and group for both native processes; confirms controller is outside their group. Signals the outer loop first and awaits its pidfd exit. Saves owner_stopped.
2. Signals the exact host SIGTERM then SIGCONT, so a job stopped by background-PTY SIGTTIN can handle termination, and awaits host exit. Checks the saved group once for unexpected survivors; never blindly kills them. Saves freeze.json phase frozen, world path, boot ID and empty group inventory. No process/endpoint polling, sleeps or stop-request counting. Host shutdown stops all plugin/agent trees and suppresses restart admission. core.restart alone is not stop-all; after the loop is gone, exit 75 cannot respawn it.
3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks without following links. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Unknown images refuse.
4. Stages the approved transformations, then checks the frozen caller guards. Both plan and apply independently require the actual frozen checkpoint, same world/current boot, dead pinned owners, empty old group and no subsequently running host for this world. A caller-supplied {remaining:0} literal is not accepted. Changed checkpoint bytes also refuse.
5. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. migration/plan.json reaches installed only after all operations complete.

Exact logical losses are only old session pending.jsonl, cancel, and unclaimed mail.deliveries rows with state='pending' and recipient not surma. Original messages, Surma copies, done rows and uncertain unknown deliveries stay; do not replay them. Loss records and original bytes/rows remain in backup/evidence. Histories/settings, scheduled SQL IDs/calendar/outcome fields, grants/requests and unrelated files remain intact.

Cron conversion: require an object and nonempty text, refuse interrupt:true, remove false/absent legacy interrupt, inner detach/notify/permanent ID, set absent kind follow-up, outer detach false. Preserve other fields and non-session JSON bytes. Review removed notify destinations; no invented answer-forwarder.

SQLite sidecars: copy the complete DB plus -journal/-wal/-shm family into immutable backup first. Recover only another disposable copy; use SQLite's backup API to stage a consistent DELETE-journal image. Plan removal of old sidecars before replacing the main DB, retaining the original physical family. This is normalization, not additional logical data loss. Tests covered committed WAL data, a genuinely hot uncommitted rollback journal, interruption and exact pre-start restore.

Full-world backup is not instantaneous: the 138,444-file / 12,441,162,164-byte / 45,310-directory size-bucket-matched synthetic replica took 70.55 s inventory + 135.39 s copy/hash verification = 205.94 s in the box. This includes repos/plugins/logs, unlike the earlier session-sized gate. It is warm-cache scale evidence, not an exact live-byte or cold-storage downtime guarantee. Allow that uncertainty in the approved maintenance plan; do not invent a timeout and kill the controller mid-copy. Power-loss recovery is not claimed.

3. Driver: ordered boot installation and pinned owner selection

The offline target boot list is:

[
  {"method":"session.manager","params":{}},
  {"method":"web.start","params":{}},
  {"method":"cred.serve","params":{}},
  {"method":"orchestrator.start","params":{}},
  {"method":"cron.start","params":{}}
]

Manager precedes web, so web-chat becomes best-effort. No mail.start deliverer remains. Resolve any extra actual boot entry before approval rather than deleting it accidentally.

The permanent <root>/target-owner exports SAUSAGE_FLAKE=path:<root>/target-selector itself, then execs the inspected loop with the exact world path. The selector's sole app is the pinned target package. <root>/old-owner similarly selects the explicit old package, never the remote default. These are permanent installation/launcher artifacts, not migration code. Their bytes/hashes are retained as manifest evidence; do not delete them with tool/.

Every later pane/cold restart uses /home/surma/sau-384-cutover/target-owner, with no caller environment required. Never invoke bare sausage-loop or the mutable gitea default for this world. Keep this command in the operator's pane-restart procedure. Rehearsal cold-launched it in the same Herdr shell without ambient SAUSAGE_FLAKE and observed the target executable. The explicit old wrapper was executed only against a separate old-backup clone.

core.restart now restarts the pinned reviewed host; it does not self-update from gitea. Future host updates require a separately approved package/closure/root/selector change. A remote push alone must never silently choose an old host against target state. Keep both GC roots through the rollback window; retire the old root only after an explicit decision, not tool deletion.

4. Driver: started → launch in the same pane → fresh verifier

The driver checks the pinned shell still owns the selected pane and no unexpected foreground process exists. It checkpoints started before any launch attempt, then submits the permanent target-owner path once via herdr pane run to the manifest's pane (live w7:p6). No shell reconstruction or in-world mediator action is needed during downtime.

Readiness is event driven: arm a kernel inotify watch on state/session/manager/registrations, queue one session.subscribe command in the console, and await its unique registration file publication. Then await actual gateway RPC completion (cred.status locked and registration acknowledgement). Herdr wait-output is internally polling and is not used by the final driver. It was an unsuccessful preliminary approach only.

Finally the driver publishes the manifest's new explicit default verification message with its stable UUID and records awaiting_verification. The manager, not the driver, runs it. This names the new mediator as verifier without claiming the cut default run was recovered. The test produced one run-start and mailbox-consumed pair for that message. Publication is not immediate model execution: in the last gate it waited 23.753 s to run-start / 24.530 s to consumption behind initial reconciliation. Do not confuse that with publication latency or a completed startup scan.

No automatic relaunch, whole-world restore or reapply is allowed after started. If launch/acknowledgement is lost, inspect the pane/checkpoint once before any repair. Do not launch a second owner blindly.

5. New mediator: verify → release web → delete tool

The explicit new verification run reads the manifest, driver.json, migration/loss/audit evidence and this runbook. Keep ingress closed until checks pass:

• one target manager and one runner ownership lock per running session; no old owners, mail deliverer or old orchestrator service;
• cred locked, no vault/key transfer; grants/requests/settings preserved. Interrupted callers reuse their existing stable credential-request keys, not automatic retries of secret-backed mutations;
• explicit target run-start without run-end is the only automatic cut-run recovery; old finished-looking history alone is neither settlement nor permission to resume;
• durable event/mailbox catch-up and same-ID default/advisor POST replay evidence holds; only intended loss/conversion/schema/new-activity differences exist. Surface uncertain mail/tool outcomes;
• scheduled jobs and non-target state match the inventory; real new mediator/runner activity is distinguished from migration changes;
• design effective grace 5 s and separate EOF-stop 5 s remain; crash supervision is enabled. Deliberate core.stop/kill disables service registration and is not a crash-supervision test.

Prior post-record live-size samples remain manager ack 0.6078 s, registration catch-up 0.0128 s, mail.send-to-default publication 0.5376 s (integrated 0.5235 s), below existing orchestrator 3 s / cred 10 s caller budgets. Disclose integrated catch-up 12.0419 s and the new initial-scan delay above. They are synthetic/warm-cache observations, not production latency guarantees. Prior real-core watch/upgrade/all-watch readiness, ~0.10% whole-host idle duty, detached error data and real cron warning/stop evidence remain linked in the result docs.

Only now use orchestrator.release for SAU-375 to release its approved commit. Inspect that method's current typed arguments; do not invent them. Verify published asset hashes and ingress. No worker publishes anything. The rework tested approved asset hashes in temporary output, not an actual live release or physical browser.

After recording the required locked startup proof, the new mediator may need Surma to unlock the new vault through the existing UI before the approved signing grant can support the final SSH command. Plan for that explicit post-start unlock; never transfer the old vault key or ask for its password. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain awaiting_verification, keep ingress/release gated, and do not reconstruct or rerun the migration.

Write external verification.json with evidence links and the required true checks named by the driver's verify mode: one_manager, no_old_owner, locked_cred, non_target_preserved, losses_exact, catchup_and_replay, web_released_and_hashes, ingress_verified. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command:

PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py verify operator.json verification.json 2>> tool-source-and-delete.log

It marks the saved plan/driver verified, logs source hashes/source-as-log, and deletes the three migration executables and bytecode caches. Confirm tool-deletion.json and absence once. Delete any other reconstructed executable/controller copies too. Retain only manifests, backup/inventory/loss/caller-audit records and logs as migration evidence, plus the separately reviewed target/permanent launcher artifacts. Rehearsal deleted all executed migration/test controllers before exporting evidence. If verification/deletion is interrupted, inspect the verified checkpoint and delete only remaining hash-matched tool copies; do not reconstruct and rerun migration effects.

Failure/rollback boundaries

Boundary	Exact recovery rule
Waiting for arm	World untouched. Inspect the saved controller identity before cancelling; do not launch another start against existing checkpoints.
Armed/captured/owner_stopped/frozen; no target attempt	External operator can use ... driver.py resume operator.json only after the saved controller has exited. It detaches a replacement; same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed.
Interrupted planning/swap, no start attempt	Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively restore the full frozen backup coherently and launch the explicit old-owner wrapper. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion.
started or later, even if boot seems failed	No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them and plan a narrow forward repair or coordinated revert.
Session revert after target activity	Close gateway/ingress. While new manager is available, unsubscribe web-chat, verify removal, and prevent gateway re-registration while the old session binary runs. Never call web.start against that old manager or restore a registry backup that reintroduces it. Actual old-manager/gateway tests proved unsubscribe safe and ID-only re-key unsafe: the old binary ignores best_effort and replays history once.
Host reboot / changed PID identity / unexpected survivor	Refuse saved native identities; re-inventory externally. Never signal a reused PID, blindly kill a group, or infer quiescence from a stop request.

Independent plugin recovery/accounting/provider-cancellation follow-ups remain independent. Accepted first-update cuts and uncertain external effects are not silently solved by this kit, grace=5, or a successful rehearsal.