sausage/docs/SAU-384-rehearsal.md

Disposable handover rehearsal — SAU-384

Authority: /docs/design/plugin-handover/design.md, Rollout; /docs/symphony/artifacts/SAU-356/plan.md, Approval and operator gates.

Status: rehearsal complete, ready for review; not a live execution approval. Rehearsal evidence and the reviewed target bundle are under /docs/symphony/artifacts/SAU-384. No shipped plugin or host behavior changes on this branch. The temporary migration implementation and native controller were executed only in a throwaway box and deleted after the successful migration verification. Their hashes/deletion record and source-as-log remain, not an executable migration tool.

Closed integration gaps

• Round 4, run 1392: allocated-block/directory/sparse-copy space budgeting; disk-independent foreground progress/failure/pidfd-death monitor; incomplete pre-plan backup cleanup before abort writes plus low-space refusal; forward-only post-start continue with durable locked proof and stable continuation ID; guarded atomic-temporary cleanup; converter dry-run before freeze. Shared-filesystem ENOSPC (world/root/log at zero free bytes), each post-start RPC failure including lost committed send reply, real 120 s registration timeout, partial atomic crash/abort, wrapper-exec handshake/resume, sidecars and full workspace passed. Tools deleted after verified success. Source/logs, fixture failures and limits: rework-1392.md. Live plan is still approval-gated.

• Round 3, run 1389: deferred approved git 9cc9fa6f is bundled/preflighted, activated only after target cred is live and Surma unlocks. Real old/new Git + new cred loopback tests covered granted SSH, locked denial and new shared SSH. Added real-driver pre-start abort/restore/old-owner relaunch, live consistent-copy/executable-only caller guards, special-file/free-space gates, finite registration deadline/failure notification, exact pre-cut prev images and durable arm-marker crash recovery. Real partial-swap/ENOSPC/audit failures, arm interruption, final Herdr cutover/preservation/sidecars and full workspace (208 host tests) passed. Tool/controllers deleted after verification. Exact logs, selection and limits: rework-1389.md; proposed live recovery/roots-side-effect plan: runbook.

• Review rework, run 1381: one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See rework-1381.md for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). Runbook now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.

• Run 1373: after mediator installed the observed public host key, approved read-only SSH inspection identified the actual Herdr pane, outer sausage-loop, world path, native architecture, old package/derivation/source and runtime closure. The actual old Nix package/loop → stop-all → offline swap → exact new Nix release package/loop passed in the disposable live-size world. A non-parent external controller separately proved loop-first stop, interruption/resumption, stale-PID refusal and safe frozen rerun. A real core.restart after loop exit could not respawn the host. See native evidence.

• Run 1369: armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed. Hooks existed only in a disposable host copy, never the selected host artifact or committed code.

• Rollback: actual old-manager/gateway tests prove unsubscribe works; ID-only re-key does not. The old binary ignores best_effort and replays history to a renamed ID. The runbook mandates the card-authorized unsubscribe option and prevents web re-registration during a session revert. A physical browser/SSE display is not claimed.

• Final selected coherent bundle is cutover-targets-1389.tar.gz (SHA-256 31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b), 23 payloads, including deferred approved git 9cc9fa6f / 177 tests. Same Nix native package/closure, approved cred main 0db40101 build 1636 / 274 tests, other reviewed plugins/web assets and SDK src/lib.rs; no diagnostic debug host. Git is not swapped offline; Surma unlock precedes its mediator activation. Earlier bundles and runs remain historical evidence. Final exact-driver native/preservation/sidecar/failure and full-workspace gates passed again; executable/controller deletion and source-as-log are in rework-1389.md.

The final frozen inventory/caller-schema/hash check remains an operator gate at the actual approved shutdown, not something a read-only rehearsal can perform in advance. Live execution still requires Surma's go and mediator/orchestrator action; no approval request is pending.

What was built and exercised

• Host source e09c4f9f0ef6b5833d6be066771eecd9e0a1f572. Original debug native controller exercised real sausage::Host, VFS, watches, locks, service supervisor and WASM with synthetic/loopback effects. Run 1373 additionally built/rehearsed the actual Nix release wrapper/package /nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0; its empty-caller-PATH runtime check built/tested a real WASI+C plugin. Workspace suite passed again; all 105 Rust source files matched the tested tree. Neither host was released/deployed or selected by the live launcher. Native artifact/closure hashes are in evidence/native-host-manifest.json.
• Reviewed session cb844151 / SAU-409 final; orchestrator c80a5854 / SAU-408; web d883fbf2 / SAU-403; agent/board retained reviewed components. Fresh card-worktree builds: mail 1600 / 53 tests, cred 1601 / 272, cron 1602 / 24, rooms 1603 / 59, all passed. No activation. Component/source provenance and every bundled file hash are in evidence/manifest.json.
• Frontend is exactly approved commit 3d76026749a5d21722033bbfe8743ac368946b0f, retained production build from SAU-375 rework-1318. It remains publish:hold; no publication was attempted.
• World SDK runtime sources already matched the selected host SDK. The only needed source-copy delta is src/lib.rs, documenting effective grace. Preserve world-owned Cargo manifests, relocated interface, and imports. A first broad SDK-copy attempt incorrectly included repository-relative Cargo paths; comparison caught this before any live change, and the final migration used only the one-file patch. Never copy the repository's SDK Cargo manifests over the world SDK.
• Accepted SAU-362 byte snapshot: 1,491 sessions / 580,355,405 bytes, including 70,689,796-byte default history. Pristine snapshot remained unpreopened. Executable clones used synthetic model settings; their historical byte volume was retained. Source snapshot consistency is per-file, not a transactional live snapshot.

Successful checks and limits

Check	Evidence/result
Old ownership → stop-all → offline swap → target boot	Old mail.start, old session send and detached ledger processes inventoried. Native external controller awaited Host.stop, verified zero remaining processes and host exit, then backed up/transformed/swapped. Final shutdown 0.0958 s in this fixture. One target manager, no mail deliverer, cred locked.
Partial swap + rerun	Injected interruption after operation 3; hash-guarded completion and repeated installed-phase apply preserved all 4,374 non-target inventoried files. No live/new work reset.
Exact losses/preservation	Only pending.jsonl, cancel, and unclaimed pending non-Surma delivery rows selected for removal. Original mail messages, done/Surma copies and uncertain unknown deliveries retained without replay; scheduled SQL IDs, calendar/outcome fields and non-session JSON bytes retained. Synthetic grant DB bytes preserved. Loss inventory backs every removed file/row.
Offline failure boundaries	Before-start rollback restored exact bytes; conflicting outside edits refused; ambiguous legacy interrupt:true refused without mutation. After-start apply and whole-world rollback both refused.
Default/advisor HTTP replay	Three identical same-ID POSTs each, before/after consumption; exactly one answered run per session. Default returned null on repeats; advisor returned its same receipt.
Explicit open target run	A run-start/consumed record with a finished-looking assistant tail, but no run-end, recovered under the same agent-run UUID, with one start. Legacy finished-looking history alone produced no target event/run.
Real VFS/runner-boundary work	Actual SDK armed then plugin-write changed; five callback-boundary runs. Run 1369 additionally forced publication before runner armed and after its empty-mailbox exit decision while detached completion was held; no lost/duplicate inputs (disposable host delay gates, actual target WASM).
Real core detached errors	Full nested error data survived a detached core.call.
Real cron research deadline	305.427 s send→stopped in the final gate; real cron invoked warning and Stop. The 305-second test deadline is the existing 300-second warning interval plus design grace 5 seconds, not a new default. Held inference was cut, no follow-up run. Original configured research default remains 1,200 seconds.
Effective grace / crash supervision	Passing native host suite exercises real WASI uniform-grace and before/after-readiness crash-supervision fixtures. Deliberate core.kill is NOT a crash: it disables that boot service; an initial rehearsal expecting its restart failed, correctly, and was replaced by the existing crash tests.
Orchestrator upgrade/idle	Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU 0.02 s / 20.0002 s ≈ 0.10%. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Run 1369 additionally verified target acknowledgement, one owner and all 13 core watches armed after old exit.

Live-size latency after real records exist

After producing actual run-start/run-end/mailbox-consumed records, subsequent full boot measured 0.544 s from manager service-start to the next boot service-start (includes discovery/ack overhead). Narrow registration catch-up samples ranged 0.0239 s to 12.0419 s in the integrated world: the long sample coincided with other synthetic recovery/surfacing work, and the untagged sink cannot isolate attribution. Do not present this as a bounded manager-only benchmark.

The clean final live-size end-to-end gate measured manager acknowledgement 0.6078 s, registration catch-up 0.0128 s, and mail.send publication to default 0.5376 s. The integrated post-record mail sample was 0.5235 s. Both publication samples are below the existing orchestrator 3 s and cred 10 s caller budgets (SAU-400 report cites the originating SAU-366 review). These warm-cache synthetic-loopback real-core observations are not live latency guarantees. First scanning still delays first new input after the fast boot acknowledgement.

Resume verification

Run 1369 forced recompilation and repeated the complete workspace suite on the restored original host source: pass, including 208 host integration tests. All 144 source files matched the selected source snapshot. Full log /logs/box/2026-10-31/tc-asa3yr-exec-3838117964.stderr (host tests lines 64–434; equivalence line 785). New tests and their limitations are detailed in resume-1369.md; new test controllers and instrumentation were deleted after successful checks. Original bundle and production source are unchanged.

Evidence and failures

All following logs are under /logs/box/2026-10-31/:

• Full workspace pass: tc-asa3yr-exec-2320283969.stderr. First cold-cache/full-load failure: tc-asa3yr-exec-158345583.stderr, missing offline async-task and a queued-build timing test. Populated the separate SDK-probe cache; entire suite then passed unchanged.
• Final stop/swap/start: tc-asa3yr-exec-1922130515.stderr; offline fault/grant preservation: tc-asa3yr-exec-616022371.stderr.
• Final live-size HTTP/deadline gate: tc-asa3yr-exec-1243993190.stderr. Earlier malformed mail to scalar, 30-second startup observation bound and blank JSONL parser mistakes were harness errors; corrected to a sequence, existing 120-second test budget and blank-line skipping. Final gate passed.
• Open-run recovery and boundary runs: tc-asa3yr-exec-903419814.stderr (subsequent console-watch attempt failed). A console is not a running watch caller; the final SDK guest watch passed in tc-asa3yr-exec-3261552495.stderr. Same log retains the real upgrade trace and idle sample. An identical already-acknowledged registration correctly produced no second catch-up; the final probe used a fresh registration identity.
• Actual old-manager unsubscribe/re-key comparison: tc-asa3yr-exec-3307393394.stderr.
• Native artifact architecture/runtime paths: tc-asa3yr-exec-899103518.stderr; bundle build/CLI: tc-asa3yr-exec-1075679130.stderr.
• Executed temporary tool and controller as log only, followed by deletion proof: tc-asa3yr-exec-2902726782.stderr (31,475 bytes). evidence/tool-deletion.json lists removed executables. No transition code was put in a shipped plugin.

The initial broad-copy fixture and earlier failed tests are not the final kit. evidence/migration-plan.json identifies the final verified migration. Provider billing/cancellation, power loss, hung host filesystem behavior, physical browsers and independent per-plugin recovery follow-ups are not claimed complete. No live DB was opened; the accepted session snapshot was reused. No live config/SDK change, stop, activation, release, restart, push, deployment or /www write occurred.

See operator runbook for exact owner/package/selector steps, frozen-audit and approval gates. Run 1373 records the final native rehearsal and remaining unverified live-only boundaries.