Record disposable handover kit, rehearsal evidence and unresolved rollout gates

Card: SAU-384

docs/SAU-384-rehearsal.md

@@ -0,0 +1,59 @@
+# Disposable handover rehearsal — SAU-384
+
+Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover/design.md), Rollout; [/docs/symphony/artifacts/SAU-356/plan.md](/docs/symphony/artifacts/SAU-356/plan.md), Approval and operator gates.
+
+**Status: blocked, not a live execution approval.** Rehearsal evidence and the reviewed target bundle are under [/docs/symphony/artifacts/SAU-384](/docs/symphony/artifacts/SAU-384). No shipped plugin or host behavior changes on this branch. The temporary migration implementation and native controller were executed only in a throwaway box and deleted after the successful migration verification. Their hashes/deletion record and source-as-log remain, not an executable migration tool.
+
+## Required resolution before completing the kit
+
+1. Pin the actual external host launcher/service owner, stop/start interface, host architecture and runtime closure. Those are not projected into this world. The supplied local native artifact is **aarch64-linux**, dynamically linked to the Nix paths printed in the build evidence; it is NOT a verified live-host package. Do not guess a systemd unit, use `core.restart` as stop-all, or install a mismatching binary.
+2. Correct the rollback alternative: **unsubscribing works; merely re-keying the registration does not**. An actual pre-best-effort session component, reading a registration written by the target component, ignores `best_effort` and sends history to its new ID. Rehearsal: unsubscribe = one barrier callback and no gateway-shaped replay; re-key = barrier plus historical replay. The test used the actual old manager and a native callback sink, not browser SSE. Either make unsubscribe mandatory, or specify what stronger meaning of “re-key” is intended. Do not claim an ID rename is safe.
+3. Complete remaining integrated acceptance: deterministic manager armed/exit-race injection on real core (the real callback-boundary sequence and primitive watch tests are not that injection), replacement orchestrator readiness after the observed upgrade, and old-manager/gateway rollback without re-registration. The old orchestrator's exit preceded the core's replacement service-start event; the immediate process snapshot still had no new owner, so readiness is not claimed.
+
+The safe unsubscribe option can be selected without inventing compatibility code. The runbook nevertheless withholds a live-ready claim until the card's rollback wording and host installation manifest are unambiguous.
+
+## What was built and exercised
+
+- Host source `e09c4f9f0ef6b5833d6be066771eecd9e0a1f572`. Disposable Nix build, native workspace suite passed after correcting the cold SDK-probe dependency cache. Target native binary was built and stripped, not released/deployed. CLI `--version` passed. Native controller used the real `sausage::Host`, real VFS, watches, locks, service supervisor and WASM execution; model/skills/memory/ledger/push effects were synthetic or loopback only.
+- Reviewed session `cb844151` / SAU-409 final; orchestrator `c80a5854` / SAU-408; web `d883fbf2` / SAU-403; agent/board retained reviewed components. Fresh card-worktree builds: mail **1600 / 53 tests**, cred **1601 / 272**, cron **1602 / 24**, rooms **1603 / 59**, all passed. No activation. Component/source provenance and every bundled file hash are in `evidence/manifest.json`.
+- Frontend is exactly approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`, retained production build from SAU-375 rework-1318. It remains `publish:hold`; no publication was attempted.
+- World SDK runtime sources already matched the selected host SDK. The only needed source-copy delta is **`src/lib.rs`**, documenting effective grace. Preserve world-owned Cargo manifests, relocated `interface`, and imports. A first broad SDK-copy attempt incorrectly included repository-relative Cargo paths; comparison caught this before any live change, and the final migration used only the one-file patch. Never copy the repository's SDK Cargo manifests over the world SDK.
+- Accepted SAU-362 byte snapshot: **1,491 sessions / 580,355,405 bytes**, including **70,689,796-byte default history**. Pristine snapshot remained unpreopened. Executable clones used synthetic model settings; their historical byte volume was retained. Source snapshot consistency is per-file, not a transactional live snapshot.
+
+## Successful checks and limits
+
+| Check | Evidence/result |
+|---|---|
+| Old ownership → stop-all → offline swap → target boot | Old `mail.start`, old session `send` and detached ledger processes inventoried. Native external controller awaited `Host.stop`, verified zero remaining processes and host exit, then backed up/transformed/swapped. Final shutdown **0.0958 s** in this fixture. One target manager, no mail deliverer, cred locked. |
+| Partial swap + rerun | Injected interruption after operation 3; hash-guarded completion and repeated installed-phase apply preserved all **4,374 non-target inventoried files**. No live/new work reset. |
+| Exact losses/preservation | Only `pending.jsonl`, `cancel`, and unclaimed pending non-Surma delivery rows selected for removal. Original mail messages, done/Surma copies and uncertain `unknown` deliveries retained without replay; scheduled SQL IDs, calendar/outcome fields and non-session JSON bytes retained. Synthetic grant DB bytes preserved. Loss inventory backs every removed file/row. |
+| Offline failure boundaries | Before-start rollback restored exact bytes; conflicting outside edits refused; ambiguous legacy `interrupt:true` refused without mutation. After-start apply and whole-world rollback both refused. |
+| Default/advisor HTTP replay | Three identical same-ID POSTs each, before/after consumption; exactly one answered run per session. Default returned null on repeats; advisor returned its same receipt. |
+| Explicit open target run | A run-start/consumed record with a finished-looking assistant tail, but no run-end, recovered under the **same agent-run UUID**, with one start. Legacy finished-looking history alone produced no target event/run. |
+| Real VFS/runner-boundary work | Actual SDK `armed` then plugin-write `changed`; five follow-ups published at durable callback/runner-end boundaries produced five distinct runs, each consuming its one ID. Does not force the exact armed-exit seam. |
+| Real core detached errors | Full nested error `data` survived a detached `core.call`. |
+| Real cron research deadline | **305.427 s** send→stopped in the final gate; real cron invoked warning and Stop. The 305-second test deadline is the existing 300-second warning interval plus design grace 5 seconds, not a new default. Held inference was cut, no follow-up run. Original configured research default remains 1,200 seconds. |
+| Effective grace / crash supervision | Passing native host suite exercises real WASI uniform-grace and before/after-readiness crash-supervision fixtures. Deliberate `core.kill` is NOT a crash: it disables that boot service; an initial rehearsal expecting its restart failed, correctly, and was replaced by the existing crash tests. |
+| Orchestrator upgrade/idle | Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU **0.02 s / 20.0002 s ≈ 0.10%**. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Replacement readiness remains unverified. |
+
+### Live-size latency after real records exist
+
+After producing actual run-start/run-end/mailbox-consumed records, subsequent full boot measured **0.544 s** from manager service-start to the next boot service-start (includes discovery/ack overhead). Narrow registration catch-up samples ranged **0.0239 s to 12.0419 s** in the integrated world: the long sample coincided with other synthetic recovery/surfacing work, and the untagged sink cannot isolate attribution. Do not present this as a bounded manager-only benchmark.
+
+The clean final live-size end-to-end gate measured manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, and `mail.send` publication to default **0.5376 s**. The integrated post-record mail sample was **0.5235 s**. Both publication samples are below the existing orchestrator **3 s** and cred **10 s** caller budgets (SAU-400 report cites the originating SAU-366 review). These warm-cache synthetic-loopback real-core observations are not live latency guarantees. First scanning still delays first new input after the fast boot acknowledgement.
+
+## Evidence and failures
+
+All following logs are under `/logs/box/2026-10-31/`:
+
+- Full workspace pass: `tc-asa3yr-exec-2320283969.stderr`. First cold-cache/full-load failure: `tc-asa3yr-exec-158345583.stderr`, missing offline async-task and a queued-build timing test. Populated the separate SDK-probe cache; entire suite then passed unchanged.
+- Final stop/swap/start: `tc-asa3yr-exec-1922130515.stderr`; offline fault/grant preservation: `tc-asa3yr-exec-616022371.stderr`.
+- Final live-size HTTP/deadline gate: `tc-asa3yr-exec-1243993190.stderr`. Earlier malformed mail `to` scalar, 30-second startup observation bound and blank JSONL parser mistakes were harness errors; corrected to a sequence, existing 120-second test budget and blank-line skipping. Final gate passed.
+- Open-run recovery and boundary runs: `tc-asa3yr-exec-903419814.stderr` (subsequent console-watch attempt failed). A console is not a running watch caller; the final SDK guest watch passed in `tc-asa3yr-exec-3261552495.stderr`. Same log retains the real upgrade trace and idle sample. An identical already-acknowledged registration correctly produced no second catch-up; the final probe used a fresh registration identity.
+- Actual old-manager unsubscribe/re-key comparison: `tc-asa3yr-exec-3307393394.stderr`.
+- Native artifact architecture/runtime paths: `tc-asa3yr-exec-899103518.stderr`; bundle build/CLI: `tc-asa3yr-exec-1075679130.stderr`.
+- Executed temporary tool and controller **as log only**, followed by deletion proof: `tc-asa3yr-exec-2902726782.stderr` (31,475 bytes). `evidence/tool-deletion.json` lists removed executables. No transition code was put in a shipped plugin.
+
+The initial broad-copy fixture and earlier failed tests are not the final kit. `evidence/migration-plan.json` identifies the final verified migration. Provider billing/cancellation, power loss, hung host filesystem behavior, physical browsers and independent per-plugin recovery follow-ups are not claimed complete. No live DB was opened; the accepted session snapshot was reused. No live config/SDK change, stop, activation, release, restart, push, deployment or `/www` write occurred.
+
+See [operator runbook](SAU-384-runbook.md) for approval gates and the deliberately blocked live steps.

docs/SAU-384-runbook.md

@@ -0,0 +1,87 @@
+# One-off handover operator runbook — blocked draft
+
+Read [rehearsal/results](SAU-384-rehearsal.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes.
+
+## 0. Finish the installation manifest (no live mutation)
+
+Evidence root: `/docs/symphony/artifacts/SAU-384/evidence/`.
+Bundle: `/docs/symphony/artifacts/SAU-384/target-artifacts.tar.gz`.
+
+- Verify bundle hash against `bundle-sha256.txt`, then every extracted target file against `manifest.json`. This bundle contains target binaries/static assets and the one-file SDK patch, **not** a migration executable.
+- Record the actual native service owner/launcher, its synchronous stop/start commands, automatic host restart mechanism, runtime closure and native architecture. Manifest fields are deliberately null until inspected. The bundled host is a local **aarch64-linux debug-profile, stripped** build of `e09c4f9f`; it has not been established as the right live-host installation package. Stop here on a mismatch; build/rehearse the right artifact rather than substituting an untested binary.
+- Pin all old plugin hashes, full boot bytes, current SDK source hashes, caller configuration, and `/proc/*/info.json` inventory. Record active agent/tool processes and any external effects whose result is uncertain. Do not infer settlement from the last assistant text.
+- Inventory executable caller payloads in cron jobs, boot and other saved callback/action/configuration stores using their predecessor schemas. History text mentioning an old RPC is not an executable payload. The tested converter covers `jobs.data` where `method == session.send`; it does not claim a full live caller-store audit. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
+- Source references for predecessor formats: session `f9122585`, `src/main.rs` Paths (`pending.jsonl`, `cancel`) and `src/queue.rs`; cron `ba75c102`; target cron README's one-off conversion procedure. Active old component hashes are separately captured; do not equate a source reference with its compiled active binary without checking.
+- Confirm the frontend artifact is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`; keep `publish:hold` until step 5. Do not publish a worktree snapshot or release it early.
+- Resolve the rollback rule before execution: an ID-only re-key is **unsafe**. Use explicit unsubscribe, and prevent the new gateway from re-registering `web-chat` against an old session binary. A different proposed “re-key” needs its own defined semantics/rehearsal.
+
+## 1. Obtain approvals, then close ingress/ownership
+
+Surma's go must precede the live core/session/cred operation. Every push requires its own fresh go. A successful rehearsal grants neither. Record the approved manifest and any accepted first-update cuts before stopping anything.
+
+Run the one-off controller **outside the world and outside the service/process group being stopped**. A plugin or agent in this world cannot be the controller: stop-all must stop it too. Recreate the narrowly reviewed temporary executable only for this approved operation, from the logged implementation with the resolved manifest; never add compatibility/transition code to a plugin.
+
+Close the actual ingress and stop the **external host/service owner** synchronously using the pinned command. Await its shutdown/process-exit completion, not a loop over `/proc`, endpoint retry, sleep, or a set of stop requests. If an outer launcher would respawn the host, deliberately stop that owner too; do not edit boot config while agents can still run. `core.restart`/exit-75 is a restart request, not stop-all. The real core's `Host.stop` suppresses service restart admission and stops all trees in parallel. Design budgets are grace 5 seconds and separate EOF-stop 5 seconds, not arbitrary new timeouts.
+
+The rehearsal's native external controller awaited `Host.stop`, asserted its process table was empty, then awaited host exit. The live equivalent needs the actual launcher identity from step 0. **That identity is currently missing; do not invent a unit name.**
+
+## 2. Back up frozen state and plan changes
+
+Only after shutdown completion:
+
+1. Make a complete frozen backup outside the target world's mutable tree, including plugin/previous binaries, boot/caller config, SDK, session histories/settings, mail, cron, cred/grants and journals. Preserve any SQLite sidecars; do not rename a live DB/WAL trio. Inventory paths/sizes/hashes and copy source hashes; retain removed rows/files explicitly. Record the old in-flight work inventory alongside it.
+2. Produce a manifest of each proposed change with before/after hashes and staged target bytes. Never regenerate the backup over an earlier backup on retry. Unknown file images refuse rather than overwrite. The tested file replacements sync the temporary file, rename and sync its parent directory.
+3. Select only the approved queue losses: old session `pending.jsonl` and `cancel`, plus unclaimed pending session delivery rows (`state='pending'`, recipient not `surma`) in `/state/mail/mail.db`. Back them up before removal. Preserve original messages, Surma copies, done rows and uncertain claimed `unknown` rows as diagnostics; the target has no deliverer to replay them. The tested cut does not infer an unknown delivery was safe to repeat.
+4. Convert each old cron `session.send` template once: require a JSON object/nonempty text; refuse `interrupt:true`; remove false/absent legacy `interrupt`, inner `detach`, `notify`, permanent `id`; set absent kind to `follow-up`; preserve all other template metadata/settings verbatim; set outer detach false. Preserve SQL IDs and every schedule/last-run/outcome field. Update all affected jobs in one `BEGIN IMMEDIATE` transaction, validate readback/integrity, and preserve non-session job JSON bytes. Record removed notify targets for review—do not replace them with an invented answer-forwarder.
+5. Stage target host and all affected plugin components from the resolved manifest as one offline cut. Do not call live `core.activate` for piecemeal contract upgrades. Unaffected plugin state/binaries stay intact. Copy only SDK `src/lib.rs` for this selected baseline; preserve relocated world Cargo manifests, imports and `interface`. If hashes differ from this baseline, resolve the coupled source-copy set rather than applying this one-file assumption blindly.
+
+The native host/runtime installation itself remains gated on the unresolved owner/architecture/closure in step 0. The rehearsal exercised real host shutdown plus offline plugin/SDK/boot replacement, not replacement of the live native service package.
+
+## 3. Install boot/caller configuration while stopped
+
+Write the exact reviewed target boot list (JSON is valid YAML):
+
+```json
+[
+  {"method":"session.manager","params":{}},
+  {"method":"web.start","params":{}},
+  {"method":"cred.serve","params":{}},
+  {"method":"orchestrator.start","params":{}},
+  {"method":"cron.start","params":{}}
+]
+```
+
+This starts the new session manager before web, so `web-chat` becomes best-effort against the new contract. `mail.start` is removed; no session deliverer remains. Check any additional actual boot entries against the manifest rather than deleting them accidentally. Install the previously inventoried caller-config conversions at the same boundary. Keep ingress closed.
+
+Resume an interrupted **offline** swap only from its saved plan: each file must have either its before or after hash. A third image is a conflict. After all operations validate, checkpoint installed. Checkpoint **started before starting target**, conservatively disabling automatic whole-world rollback even if boot later fails.
+
+## 4. Start/reconcile and verify
+
+Start the pinned target external owner once. Observe readiness/completion events, not process/endpoint polling. Confirm:
+
+- one manager, one ownership lock per running session; no old mail deliverer, old orchestrator service or old session agent owners;
+- cred starts **locked**, with no master-key/vault handover; stored grants/requests/settings remain intact; interrupted credential request callers retry their same stable key, not a secret-backed external mutation;
+- only explicit target run-start-without-run-end work is recovered under its existing agent-run UUID; finished-looking old history alone is not resumed;
+- pending new mailbox work and durable events catch up, callbacks tolerate repeats, fresh default/advisor POST retries use the same ID, no old pending queue/undelivered mail is replayed;
+- scheduled jobs/grants/non-target state match the inventory except recorded intended schema/activity additions; retained `unknown` mail/tool outcomes remain visible;
+- effective grace is 5 seconds regardless of legacy metadata, and crash supervision has not disabled a service. Deliberate `core.stop`/`core.kill` disables that service registration; do not use it as a crash-supervision test;
+- first-update cuts are surfaced: old runs without target records cannot be recovered automatically; tools may already have external effects; matching histories do not prove an effect did or did not happen.
+
+Repeat live-size post-record publication/catch-up measurements if production timing is a release gate; compare `mail.send` to the existing 3-second orchestrator / 10-second cred limits. Fast manager boot acknowledgement is not a completed initial scan. Do not turn catch-up into a timer/poll loop.
+
+## 5. Release web, reopen ingress, delete the one-off executable
+
+Only after session and web plugins are active and verified, mediator/orchestrator uses the existing **`orchestrator.release` workflow for SAU-375** to clear its `publish:hold` and publish its approved commit. Check that method's current typed arguments before execution; this draft intentionally does not invent them. No worker performs this action.
+
+Verify the published approved asset hashes and ingress once, mark the migration verified, then **delete its executable and any executable cached/controller copies**. The rehearsal did this; live mediator must too. Retain only the manifest, backup/inventory/loss/caller-conversion records and logs as migration evidence, plus separately reviewed target artifacts. Do not retain an enabled migration plugin or boot entry.
+
+## Failure/rollback boundaries
+
+| Boundary | Action |
+|---|---|
+| Before offline mutation | Stop with evidence; old frozen world is untouched. |
+| During swap, target never started | Resume the saved hash-guarded plan, or restore the full frozen backup and old external owner/configuration coherently. Do not partially start mixed contracts. |
+| Any target start attempted | No automatic full restore/reapply. New histories, mail, jobs, grants or external effects may already exist. Preserve new state; diagnose and plan a narrow forward repair/revert. Never blindly rerun an external effect or undo new work. |
+| Session revert after target activity | A new approved operation: close gateway/ingress, `session.unsubscribe {id:"web-chat"}` while the new manager is still available, verify removal, prevent gateway re-registration while old session runs, then coordinate the stopped version change. Do not restore a registry backup that reintroduces the registration. ID-only rename is proven unsafe. Full actual gateway revert remains unverified. |
+
+Independent plugin recovery follow-ups remain independent. Neither uniform grace nor this one-off migration proves their external side effects, usage accounting, crash recovery or provider cancellation are solved.