sausage/docs/SAU-384-rehearsal.md

# Disposable handover rehearsal — SAU-384

Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover/design.md), Rollout; [/docs/symphony/artifacts/SAU-356/plan.md](/docs/symphony/artifacts/SAU-356/plan.md), Approval and operator gates.

**Status: rehearsal complete, ready for review; not a live execution approval.** Rehearsal evidence and the reviewed target bundle are under [/docs/symphony/artifacts/SAU-384](/docs/symphony/artifacts/SAU-384). No shipped plugin or host behavior changes on this branch. The temporary migration implementation and native controller were executed only in a throwaway box and deleted after the successful migration verification. Their hashes/deletion record and source-as-log remain, not an executable migration tool.

## Closed integration gaps

- **Round 4, run 1392:** allocated-block/directory/sparse-copy space budgeting; disk-independent foreground progress/failure/pidfd-death monitor; incomplete pre-plan backup cleanup before abort writes plus low-space refusal; forward-only post-start continue with durable locked proof and stable continuation ID; guarded atomic-temporary cleanup; converter dry-run before freeze. Shared-filesystem ENOSPC (world/root/log at zero free bytes), each post-start RPC failure including lost committed send reply, real 120 s registration timeout, partial atomic crash/abort, wrapper-exec handshake/resume, sidecars and full workspace passed. Tools deleted after verified success. Source/logs, fixture failures and limits: [rework-1392.md](rework-1392.md). Live plan is still approval-gated.

- **Round 3, run 1389:** deferred approved git 9cc9fa6f is bundled/preflighted, activated only after target cred is live and Surma unlocks. Real old/new Git + new cred loopback tests covered granted SSH, locked denial and new shared SSH. Added real-driver pre-start abort/restore/old-owner relaunch, live consistent-copy/executable-only caller guards, special-file/free-space gates, finite registration deadline/failure notification, exact pre-cut prev images and durable arm-marker crash recovery. Real partial-swap/ENOSPC/audit failures, arm interruption, final Herdr cutover/preservation/sidecars and full workspace (208 host tests) passed. Tool/controllers deleted after verification. Exact logs, selection and limits: [rework-1389.md](rework-1389.md); proposed live recovery/roots-side-effect plan: [runbook](runbook.md).

- **Review rework, run 1381:** one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See [rework-1381.md](rework-1381.md) for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). [Runbook](runbook.md) now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.
- **Run 1373:** after mediator installed the observed public host key, approved read-only SSH inspection identified the actual Herdr pane, outer `sausage-loop`, world path, native architecture, old package/derivation/source and runtime closure. The actual old Nix package/loop → stop-all → offline swap → exact new Nix release package/loop passed in the disposable live-size world. A non-parent external controller separately proved loop-first stop, interruption/resumption, stale-PID refusal and safe frozen rerun. A real core.restart after loop exit could not respawn the host. See [native evidence](resume-1373.md).
- **Run 1369:** armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed. Hooks existed only in a disposable host copy, never the selected host artifact or committed code.
- **Rollback:** actual old-manager/gateway tests prove **unsubscribe works; ID-only re-key does not**. The old binary ignores best_effort and replays history to a renamed ID. The runbook mandates the card-authorized unsubscribe option and prevents web re-registration during a session revert. A physical browser/SSE display is not claimed.
- Final selected coherent bundle is **`cutover-targets-1389.tar.gz`** (SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`), **23 payloads**, including deferred approved **git 9cc9fa6f / 177 tests**. Same Nix native package/closure, approved cred main `0db40101` build **1636 / 274 tests**, other reviewed plugins/web assets and SDK `src/lib.rs`; no diagnostic debug host. Git is not swapped offline; Surma unlock precedes its mediator activation. Earlier bundles and runs remain historical evidence. Final exact-driver native/preservation/sidecar/failure and full-workspace gates passed again; executable/controller deletion and source-as-log are in rework-1389.md.

The final frozen inventory/caller-schema/hash check remains an operator gate at the actual approved shutdown, not something a read-only rehearsal can perform in advance. Live execution still requires Surma's go and mediator/orchestrator action; no approval request is pending.

## What was built and exercised

- Host source `e09c4f9f0ef6b5833d6be066771eecd9e0a1f572`. Original debug native controller exercised real `sausage::Host`, VFS, watches, locks, service supervisor and WASM with synthetic/loopback effects. Run 1373 additionally built/rehearsed the actual **Nix release wrapper/package** `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; its empty-caller-PATH runtime check built/tested a real WASI+C plugin. Workspace suite passed again; all 105 Rust source files matched the tested tree. Neither host was released/deployed or selected by the live launcher. Native artifact/closure hashes are in `evidence/native-host-manifest.json`.
- Reviewed session `cb844151` / SAU-409 final; orchestrator `c80a5854` / SAU-408; web `d883fbf2` / SAU-403; agent/board retained reviewed components. Fresh card-worktree builds: mail **1600 / 53 tests**, cred **1601 / 272**, cron **1602 / 24**, rooms **1603 / 59**, all passed. No activation. Component/source provenance and every bundled file hash are in `evidence/manifest.json`.
- Frontend is exactly approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`, retained production build from SAU-375 rework-1318. It remains `publish:hold`; no publication was attempted.
- World SDK runtime sources already matched the selected host SDK. The only needed source-copy delta is **`src/lib.rs`**, documenting effective grace. Preserve world-owned Cargo manifests, relocated `interface`, and imports. A first broad SDK-copy attempt incorrectly included repository-relative Cargo paths; comparison caught this before any live change, and the final migration used only the one-file patch. Never copy the repository's SDK Cargo manifests over the world SDK.
- Accepted SAU-362 byte snapshot: **1,491 sessions / 580,355,405 bytes**, including **70,689,796-byte default history**. Pristine snapshot remained unpreopened. Executable clones used synthetic model settings; their historical byte volume was retained. Source snapshot consistency is per-file, not a transactional live snapshot.

## Successful checks and limits

| Check | Evidence/result |
|---|---|
| Old ownership → stop-all → offline swap → target boot | Old `mail.start`, old session `send` and detached ledger processes inventoried. Native external controller awaited `Host.stop`, verified zero remaining processes and host exit, then backed up/transformed/swapped. Final shutdown **0.0958 s** in this fixture. One target manager, no mail deliverer, cred locked. |
| Partial swap + rerun | Injected interruption after operation 3; hash-guarded completion and repeated installed-phase apply preserved all **4,374 non-target inventoried files**. No live/new work reset. |
| Exact losses/preservation | Only `pending.jsonl`, `cancel`, and unclaimed pending non-Surma delivery rows selected for removal. Original mail messages, done/Surma copies and uncertain `unknown` deliveries retained without replay; scheduled SQL IDs, calendar/outcome fields and non-session JSON bytes retained. Synthetic grant DB bytes preserved. Loss inventory backs every removed file/row. |
| Offline failure boundaries | Before-start rollback restored exact bytes; conflicting outside edits refused; ambiguous legacy `interrupt:true` refused without mutation. After-start apply and whole-world rollback both refused. |
| Default/advisor HTTP replay | Three identical same-ID POSTs each, before/after consumption; exactly one answered run per session. Default returned null on repeats; advisor returned its same receipt. |
| Explicit open target run | A run-start/consumed record with a finished-looking assistant tail, but no run-end, recovered under the **same agent-run UUID**, with one start. Legacy finished-looking history alone produced no target event/run. |
| Real VFS/runner-boundary work | Actual SDK `armed` then plugin-write `changed`; five callback-boundary runs. Run 1369 additionally forced publication before runner `armed` and after its empty-mailbox exit decision while detached completion was held; no lost/duplicate inputs (disposable host delay gates, actual target WASM). |
| Real core detached errors | Full nested error `data` survived a detached `core.call`. |
| Real cron research deadline | **305.427 s** send→stopped in the final gate; real cron invoked warning and Stop. The 305-second test deadline is the existing 300-second warning interval plus design grace 5 seconds, not a new default. Held inference was cut, no follow-up run. Original configured research default remains 1,200 seconds. |
| Effective grace / crash supervision | Passing native host suite exercises real WASI uniform-grace and before/after-readiness crash-supervision fixtures. Deliberate `core.kill` is NOT a crash: it disables that boot service; an initial rehearsal expecting its restart failed, correctly, and was replaced by the existing crash tests. |
| Orchestrator upgrade/idle | Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU **0.02 s / 20.0002 s ≈ 0.10%**. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Run 1369 additionally verified target acknowledgement, one owner and all 13 core watches armed after old exit. |

### Live-size latency after real records exist

After producing actual run-start/run-end/mailbox-consumed records, subsequent full boot measured **0.544 s** from manager service-start to the next boot service-start (includes discovery/ack overhead). Narrow registration catch-up samples ranged **0.0239 s to 12.0419 s** in the integrated world: the long sample coincided with other synthetic recovery/surfacing work, and the untagged sink cannot isolate attribution. Do not present this as a bounded manager-only benchmark.

The clean final live-size end-to-end gate measured manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, and `mail.send` publication to default **0.5376 s**. The integrated post-record mail sample was **0.5235 s**. Both publication samples are below the existing orchestrator **3 s** and cred **10 s** caller budgets (SAU-400 report cites the originating SAU-366 review). These warm-cache synthetic-loopback real-core observations are not live latency guarantees. First scanning still delays first new input after the fast boot acknowledgement.

## Resume verification

Run 1369 forced recompilation and repeated the complete workspace suite on the restored original host source: pass, including **208 host integration tests**. All **144 source files** matched the selected source snapshot. Full log `/logs/box/2026-10-31/tc-asa3yr-exec-3838117964.stderr` (host tests lines 64–434; equivalence line 785). New tests and their limitations are detailed in `resume-1369.md`; new test controllers and instrumentation were deleted after successful checks. Original bundle and production source are unchanged.

## Evidence and failures

All following logs are under `/logs/box/2026-10-31/`:

- Full workspace pass: `tc-asa3yr-exec-2320283969.stderr`. First cold-cache/full-load failure: `tc-asa3yr-exec-158345583.stderr`, missing offline async-task and a queued-build timing test. Populated the separate SDK-probe cache; entire suite then passed unchanged.
- Final stop/swap/start: `tc-asa3yr-exec-1922130515.stderr`; offline fault/grant preservation: `tc-asa3yr-exec-616022371.stderr`.
- Final live-size HTTP/deadline gate: `tc-asa3yr-exec-1243993190.stderr`. Earlier malformed mail `to` scalar, 30-second startup observation bound and blank JSONL parser mistakes were harness errors; corrected to a sequence, existing 120-second test budget and blank-line skipping. Final gate passed.
- Open-run recovery and boundary runs: `tc-asa3yr-exec-903419814.stderr` (subsequent console-watch attempt failed). A console is not a running watch caller; the final SDK guest watch passed in `tc-asa3yr-exec-3261552495.stderr`. Same log retains the real upgrade trace and idle sample. An identical already-acknowledged registration correctly produced no second catch-up; the final probe used a fresh registration identity.
- Actual old-manager unsubscribe/re-key comparison: `tc-asa3yr-exec-3307393394.stderr`.
- Native artifact architecture/runtime paths: `tc-asa3yr-exec-899103518.stderr`; bundle build/CLI: `tc-asa3yr-exec-1075679130.stderr`.
- Executed temporary tool and controller **as log only**, followed by deletion proof: `tc-asa3yr-exec-2902726782.stderr` (31,475 bytes). `evidence/tool-deletion.json` lists removed executables. No transition code was put in a shipped plugin.

The initial broad-copy fixture and earlier failed tests are not the final kit. `evidence/migration-plan.json` identifies the final verified migration. Provider billing/cancellation, power loss, hung host filesystem behavior, physical browsers and independent per-plugin recovery follow-ups are not claimed complete. No live DB was opened; the accepted session snapshot was reused. No live config/SDK change, stop, activation, release, restart, push, deployment or `/www` write occurred.

See [operator runbook](runbook.md) for exact owner/package/selector steps, frozen-audit and approval gates. [Run 1373](resume-1373.md) records the final native rehearsal and remaining unverified live-only boundaries.