# One-off handover operator runbook — blocked draft Read [rehearsal/results](SAU-384-rehearsal.md) and [resume verification](SAU-384-resume-1369.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes. ## 0. Finish the installation manifest (no live mutation) Evidence root: `/docs/symphony/artifacts/SAU-384/evidence/`. Bundle: `/docs/symphony/artifacts/SAU-384/target-artifacts.tar.gz`. - Verify bundle hash against `bundle-sha256.txt`, then every extracted target file against `manifest.json`. This bundle contains target binaries/static assets and the one-file SDK patch, **not** a migration executable. - Record the actual native service owner/launcher, its synchronous stop/start commands, automatic host restart mechanism, runtime closure and native architecture. Manifest fields are deliberately null until inspected. The bundled host is a local **aarch64-linux debug-profile, stripped** build of `e09c4f9f`; it has not been established as the right live-host installation package. Stop here on a mismatch; build/rehearse the right artifact rather than substituting an untested binary. - Pin all old plugin hashes, full boot bytes, current SDK source hashes, caller configuration, and `/proc/*/info.json` inventory. Record active agent/tool processes and any external effects whose result is uncertain. Do not infer settlement from the last assistant text. - Inventory executable caller payloads in cron jobs, boot and other saved callback/action/configuration stores using their predecessor schemas. History text mentioning an old RPC is not an executable payload. The tested converter covers `jobs.data` where `method == session.send`; it does not claim a full live caller-store audit. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution. - Source references for predecessor formats: session `f9122585`, `src/main.rs` Paths (`pending.jsonl`, `cancel`) and `src/queue.rs`; cron `ba75c102`; target cron README's one-off conversion procedure. Active old component hashes are separately captured; do not equate a source reference with its compiled active binary without checking. - Confirm the frontend artifact is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`; keep `publish:hold` until step 5. Do not publish a worktree snapshot or release it early. - Rollback uses the card's explicit **unsubscribe** option. ID-only re-key is **unsafe**, proven again with the actual old manager and gateway in run 1369. Prevent the gateway from re-registering `web-chat` against an old session binary; never call `web.start` in that state. A different proposed “re-key” needs its own defined semantics/rehearsal. ## 1. Obtain approvals, then close ingress/ownership Surma's go must precede the live core/session/cred operation. Every push requires its own fresh go. A successful rehearsal grants neither. Record the approved manifest and any accepted first-update cuts before stopping anything. Run the one-off controller **outside the world and outside the service/process group being stopped**. A plugin or agent in this world cannot be the controller: stop-all must stop it too. Recreate the narrowly reviewed temporary executable only for this approved operation, from the logged implementation with the resolved manifest; never add compatibility/transition code to a plugin. Close the actual ingress and stop the **external host/service owner** synchronously using the pinned command. Await its shutdown/process-exit completion, not a loop over `/proc`, endpoint retry, sleep, or a set of stop requests. If an outer launcher would respawn the host, deliberately stop that owner too; do not edit boot config while agents can still run. `core.restart`/exit-75 is a restart request, not stop-all. The real core's `Host.stop` suppresses service restart admission and stops all trees in parallel. Design budgets are grace 5 seconds and separate EOF-stop 5 seconds, not arbitrary new timeouts. The rehearsal's native external controller awaited `Host.stop`, asserted its process table was empty, then awaited host exit. The live equivalent needs the actual launcher identity from step 0. **That identity is currently missing; do not invent a unit name.** Source config has `services.sausage.enable = false` and describes a Herdr panel; read-only systemd-web inspection corroborates a user Herdr socket, not a system `sausage.service`. Credential request `055995a51608c57dd25b323bea731fcb` awaits approval for read-only inspection to pin the actual panel/launcher. It is not permission to stop or change it. ## 2. Back up frozen state and plan changes Only after shutdown completion: 1. Make a complete frozen backup outside the target world's mutable tree, including plugin/previous binaries, boot/caller config, SDK, session histories/settings, mail, cron, cred/grants and journals. Preserve any SQLite sidecars; do not rename a live DB/WAL trio. Inventory paths/sizes/hashes and copy source hashes; retain removed rows/files explicitly. Record the old in-flight work inventory alongside it. 2. Produce a manifest of each proposed change with before/after hashes and staged target bytes. Never regenerate the backup over an earlier backup on retry. Unknown file images refuse rather than overwrite. The tested file replacements sync the temporary file, rename and sync its parent directory. 3. Select only the approved queue losses: old session `pending.jsonl` and `cancel`, plus unclaimed pending session delivery rows (`state='pending'`, recipient not `surma`) in `/state/mail/mail.db`. Back them up before removal. Preserve original messages, Surma copies, done rows and uncertain claimed `unknown` rows as diagnostics; the target has no deliverer to replay them. The tested cut does not infer an unknown delivery was safe to repeat. 4. Convert each old cron `session.send` template once: require a JSON object/nonempty text; refuse `interrupt:true`; remove false/absent legacy `interrupt`, inner `detach`, `notify`, permanent `id`; set absent kind to `follow-up`; preserve all other template metadata/settings verbatim; set outer detach false. Preserve SQL IDs and every schedule/last-run/outcome field. Update all affected jobs in one `BEGIN IMMEDIATE` transaction, validate readback/integrity, and preserve non-session job JSON bytes. Record removed notify targets for review—do not replace them with an invented answer-forwarder. 5. Stage target host and all affected plugin components from the resolved manifest as one offline cut. Do not call live `core.activate` for piecemeal contract upgrades. Unaffected plugin state/binaries stay intact. Copy only SDK `src/lib.rs` for this selected baseline; preserve relocated world Cargo manifests, imports and `interface`. If hashes differ from this baseline, resolve the coupled source-copy set rather than applying this one-file assumption blindly. The native host/runtime installation itself remains gated on the unresolved owner/architecture/closure in step 0. The rehearsal exercised real host shutdown plus offline plugin/SDK/boot replacement, not replacement of the live native service package. ## 3. Install boot/caller configuration while stopped Write the exact reviewed target boot list (JSON is valid YAML): ```json [ {"method":"session.manager","params":{}}, {"method":"web.start","params":{}}, {"method":"cred.serve","params":{}}, {"method":"orchestrator.start","params":{}}, {"method":"cron.start","params":{}} ] ``` This starts the new session manager before web, so `web-chat` becomes best-effort against the new contract. `mail.start` is removed; no session deliverer remains. Check any additional actual boot entries against the manifest rather than deleting them accidentally. Install the previously inventoried caller-config conversions at the same boundary. Keep ingress closed. Resume an interrupted **offline** swap only from its saved plan: each file must have either its before or after hash. A third image is a conflict. After all operations validate, checkpoint installed. Checkpoint **started before starting target**, conservatively disabling automatic whole-world rollback even if boot later fails. ## 4. Start/reconcile and verify Start the pinned target external owner once. Observe readiness/completion events, not process/endpoint polling. Confirm: - one manager, one ownership lock per running session; no old mail deliverer, old orchestrator service or old session agent owners; - cred starts **locked**, with no master-key/vault handover; stored grants/requests/settings remain intact; interrupted credential request callers retry their same stable key, not a secret-backed external mutation; - only explicit target run-start-without-run-end work is recovered under its existing agent-run UUID; finished-looking old history alone is not resumed; - pending new mailbox work and durable events catch up, callbacks tolerate repeats, fresh default/advisor POST retries use the same ID, no old pending queue/undelivered mail is replayed; - scheduled jobs/grants/non-target state match the inventory except recorded intended schema/activity additions; retained `unknown` mail/tool outcomes remain visible; - effective grace is 5 seconds regardless of legacy metadata, and crash supervision has not disabled a service. Deliberate `core.stop`/`core.kill` disables that service registration; do not use it as a crash-supervision test; - first-update cuts are surfaced: old runs without target records cannot be recovered automatically; tools may already have external effects; matching histories do not prove an effect did or did not happen. Repeat live-size post-record publication/catch-up measurements if production timing is a release gate; compare `mail.send` to the existing 3-second orchestrator / 10-second cred limits. Fast manager boot acknowledgement is not a completed initial scan. Do not turn catch-up into a timer/poll loop. ## 5. Release web, reopen ingress, delete the one-off executable Only after session and web plugins are active and verified, mediator/orchestrator uses the existing **`orchestrator.release` workflow for SAU-375** to clear its `publish:hold` and publish its approved commit. Check that method's current typed arguments before execution; this draft intentionally does not invent them. No worker performs this action. Verify the published approved asset hashes and ingress once, mark the migration verified, then **delete its executable and any executable cached/controller copies**. The rehearsal did this; live mediator must too. Retain only the manifest, backup/inventory/loss/caller-conversion records and logs as migration evidence, plus separately reviewed target artifacts. Do not retain an enabled migration plugin or boot entry. ## Failure/rollback boundaries | Boundary | Action | |---|---| | Before offline mutation | Stop with evidence; old frozen world is untouched. | | During swap, target never started | Resume the saved hash-guarded plan, or restore the full frozen backup and old external owner/configuration coherently. Do not partially start mixed contracts. | | Any target start attempted | No automatic full restore/reapply. New histories, mail, jobs, grants or external effects may already exist. Preserve new state; diagnose and plan a narrow forward repair/revert. Never blindly rerun an external effect or undo new work. | | Session revert after target activity | A new approved operation: close gateway/ingress, `session.unsubscribe {id:"web-chat"}` while the new manager is still available, verify removal, prevent gateway re-registration while old session runs, then coordinate the stopped version change. Do not restore a registry backup that reintroduces the registration. ID-only rename is proven unsafe. Run 1369 verified this with the actual gateway left running and no re-registration: zero historical callbacks after unsubscribe; ID-only re-key replayed history. Physical browser/SSE display is not claimed. | Independent plugin recovery follow-ups remain independent. Neither uniform grace nor this one-off migration proves their external side effects, usage accounting, crash recovery or provider cancellation are solved.