97 lines
16 KiB
Markdown
97 lines
16 KiB
Markdown
# One-off handover operator runbook — rehearsed, approval required
|
|
|
|
Read [rehearsal/results](implementation.md), [race/rollback verification](resume-1369.md), [native owner/package rehearsal](resume-1373.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes.
|
|
|
|
## 0. Verify and freeze the installation manifest (read-only first)
|
|
|
|
Evidence root: `/docs/symphony/artifacts/SAU-384/evidence/`.
|
|
Selected coherent bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`.
|
|
The older `target-artifacts.tar.gz` remains prior rehearsal evidence, **not** the native installation selection.
|
|
|
|
- Verify the new bundle hash against `evidence/cutover-targets-sha256.txt`, then every extracted file against `evidence/manifest.json`. It contains reviewed plugins/static assets, SDK `src/lib.rs`, and `native-package.nar.gz` with its closure manifest, **not** a migration executable. It excludes the old diagnostic debug host.
|
|
- Native selection is the Nix release package `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux, built from this card's tree with unchanged host source `e09c4f9f`. `evidence/native-host-manifest.json` pins its NAR/file hashes and every runtime closure path/hash. All dependency paths match the observed old package's closure; only the host root differs. Recheck closure availability/hashes before execution. Mediator may import the exact reviewed NAR if needed (`gzip -dc native-package.nar.gz | nix-store --import`); never substitute the debug binary. Import/deployment is not a worker action.
|
|
- Owner is **Herdr workspace w7, pane w7:p6 (tab w7:t5)** running `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`, not a systemd Sausage unit. Re-identify its current loop/host PIDs using read-only `herdr pane process-info --pane w7:p6`; observed PIDs in `resume-1373-inspection.json` are evidence, never reusable constants. The loop refreshes a remote flake after any nonzero exit, including core.restart (75). Its existing 2-second restart sleep is not a migration quiescence mechanism.
|
|
- Pin all old plugin hashes, full boot bytes, current SDK source hashes, caller configuration, and `/proc/*/info.json` inventory. Record active agent/tool processes and any external effects whose result is uncertain. Do not infer settlement from the last assistant text.
|
|
- Inventory executable caller payloads using predecessor schemas, not substring rewriting. Run 1373 read-only audit covered cron jobs; board hooks, invariant target, attempt snapshots, subscriptions/batches; room/workflow events; orchestrator runs, skill reviews and host jobs; manager registrations and configuration. `evidence/resume-1373-inspection.json` records six legacy cron templates requiring conversion, an already-target-valid board hook **and invariant target**, eight historical running hook snapshots that must NOT replay, zero subscription batches, one abandoned room with no pending work, and no live session manager registrations. The only observed conversion beyond boot is cron `jobs.data` where `method == session.send`. Repeat against the frozen bytes: this observation is not a transactional final snapshot. Prompt strings/diagnostics are not executable templates; preserve them verbatim. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
|
- Source references for predecessor formats: session `f9122585`, `src/main.rs` Paths (`pending.jsonl`, `cancel`) and `src/queue.rs`; cron `ba75c102`; target cron README's one-off conversion procedure. Active old component hashes are separately captured; do not equate a source reference with its compiled active binary without checking.
|
|
- Confirm the frontend artifact is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`; keep `publish:hold` until step 5. Do not publish a worktree snapshot or release it early.
|
|
- Rollback uses the card's explicit **unsubscribe** option. ID-only re-key is **unsafe**, proven again with the actual old manager and gateway in run 1369. Prevent the gateway from re-registering `web-chat` against an old session binary; never call `web.start` in that state. A different proposed “re-key” needs its own defined semantics/rehearsal.
|
|
|
|
## 1. Obtain approvals, then close ingress/ownership
|
|
|
|
Surma's go must precede the live core/session/cred operation. Every push requires its own fresh go. A successful rehearsal grants neither. Record the approved manifest and any accepted first-update cuts before stopping anything.
|
|
|
|
Run the one-off controller **outside the world and outside the service/process group being stopped**. A plugin or agent in this world cannot be the controller: stop-all must stop it too. Recreate the narrowly reviewed temporary executable only for this approved operation, from the logged implementation with the resolved manifest; never add compatibility/transition code to a plugin.
|
|
|
|
Close ingress before stopping ownership. The tested exact native boundary is the deleted one-off `native_freeze.py` (source-as-log linked in resume-1373): invoke its reconstructed, reviewed copy **outside the world/group**, with current loop PID, current host PID and an external checkpoint path. Use external root `/home/surma/sau-384-cutover`, with the reconstructed scripts in `tool/` and checkpoint `evidence/freeze.json`. The observed Python interpreter is `/nix/store/33i3kijbra0cffncn3rmi9bdrav7a3y3-python3-3.13.15-env/bin/python3`. After Surma's go and fresh PID/identity inventory, the exact stop invocation is `PYTHONDONTWRITEBYTECODE=1 <that-python> tool/native_freeze.py "$LOOP_PID" "$HOST_PID" evidence/freeze.json` from the external root. Do not pass the rehearsal-only failure-injection argument. It pins boot ID/start ticks/executable/group and pidfds, sends SIGTERM to the **loop first**, awaits loop exit, checkpoints `owner_stopped`, then sends SIGTERM to the exact host and awaits its exit. Host SIGTERM awaits real `Host.stop`; its old source was inspected at the actual Nix derivation. After both exits, inspect the saved group once; any unexpected survivor blocks without blindly killing it. No process polling, endpoint retry, sleep or service-stop-request counting. Do not close the Herdr server or unrelated panes.
|
|
|
|
The non-parent-controller rehearsal proves loop exit does not kill the migration controller or old host prematurely; the host survives its parent and drains on its own signal. Interrupting after loop exit is resumable from the same pinned checkpoint; stale PID identities refuse. Even a real core.restart after loop exit cannot respawn the host. `core.restart` by itself is **not** stop-all. Core's shutdown suppresses service restart admission and stops all trees in parallel. Design budgets remain grace 5 seconds and separate EOF-stop 5 seconds; the controller awaits completion rather than inventing a live timeout. Never edit world state before the `frozen` checkpoint.
|
|
|
|
## 2. Back up frozen state and plan changes
|
|
|
|
Only after shutdown completion:
|
|
|
|
1. Make a complete frozen backup outside the target world's mutable tree, including plugin/previous binaries, boot/caller config, SDK, session histories/settings, mail, cron, cred/grants and journals. Preserve any SQLite sidecars; do not rename a live DB/WAL trio. Inventory paths/sizes/hashes and copy source hashes; retain removed rows/files explicitly. Record the old in-flight work inventory alongside it.
|
|
2. Produce a manifest of each proposed change with before/after hashes and staged target bytes. Never regenerate the backup over an earlier backup on retry. Unknown file images refuse rather than overwrite. The tested file replacements sync the temporary file, rename and sync its parent directory.
|
|
3. Select only the approved queue losses: old session `pending.jsonl` and `cancel`, plus unclaimed pending session delivery rows (`state='pending'`, recipient not `surma`) in `/state/mail/mail.db`. Back them up before removal. Preserve original messages, Surma copies, done rows and uncertain claimed `unknown` rows as diagnostics; the target has no deliverer to replay them. The tested cut does not infer an unknown delivery was safe to repeat.
|
|
4. Convert each old cron `session.send` template once: require a JSON object/nonempty text; refuse `interrupt:true`; remove false/absent legacy `interrupt`, inner `detach`, `notify`, permanent `id`; set absent kind to `follow-up`; preserve all other template metadata/settings verbatim; set outer detach false. Preserve SQL IDs and every schedule/last-run/outcome field. Update all affected jobs in one `BEGIN IMMEDIATE` transaction, validate readback/integrity, and preserve non-session job JSON bytes. Record removed notify targets for review—do not replace them with an invented answer-forwarder.
|
|
5. Stage target host and all affected plugin components from the resolved manifest as one offline cut. Do not call live `core.activate` for piecemeal contract upgrades. Unaffected plugin state/binaries stay intact. Copy only SDK `src/lib.rs` for this selected baseline; preserve relocated world Cargo manifests, imports and `interface`. If hashes differ from this baseline, resolve the coupled source-copy set rather than applying this one-file assumption blindly.
|
|
|
|
Native host installation is an **outer launcher selection**, never an overwrite of immutable `/nix/store`. While stopped, install the static selector flake recorded in the manifest at `/home/surma/sau-384-cutover/target-selector/flake.nix` (outside the world). The proposed parent was absent at read-only inspection; refuse an unexpected existing image rather than overwrite it. Its sole app points to the exact target package's `bin/sausage`. Hash-guard and back up any existing selector; do not overwrite an unknown image. The selector is target launcher configuration, not a migration executable. The actual old Nix package/loop → offline plugin/SDK/boot swap → new Nix package/loop was rehearsed; neither public remote refresh nor a source-only host was substituted.
|
|
|
|
## 3. Install boot/caller configuration while stopped
|
|
|
|
Write the exact reviewed target boot list (JSON is valid YAML):
|
|
|
|
```json
|
|
[
|
|
{"method":"session.manager","params":{}},
|
|
{"method":"web.start","params":{}},
|
|
{"method":"cred.serve","params":{}},
|
|
{"method":"orchestrator.start","params":{}},
|
|
{"method":"cron.start","params":{}}
|
|
]
|
|
```
|
|
|
|
This starts the new session manager before web, so `web-chat` becomes best-effort against the new contract. `mail.start` is removed; no session deliverer remains. Check any additional actual boot entries against the manifest rather than deleting them accidentally. Install the previously inventoried caller-config conversions at the same boundary. Keep ingress closed.
|
|
|
|
Resume an interrupted **offline** swap only from its saved plan: each file must have either its before or after hash. A third image is a conflict. After all operations validate, checkpoint installed. Checkpoint **started before starting target**, conservatively disabling automatic whole-world rollback even if boot later fails.
|
|
|
|
## 4. Start/reconcile and verify
|
|
|
|
Checkpoint `started` **before any launch attempt**. In the identified Herdr shell, mediator starts the same inspected loop exactly once with an explicit pinned selector:
|
|
|
|
```sh
|
|
SAUSAGE_FLAKE=path:/home/surma/sau-384-cutover/target-selector \
|
|
/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/
|
|
```
|
|
|
|
The selector directory must contain exactly the recorded installation bytes from step 2, not a mutable public flake. This pins crash restarts to the same reviewed host instead of refreshing unknown future code. Keep that selector/closure reachable for the target owner after deleting the migration tool. Observe boot/RPC/watch completion events, not process/endpoint polling. If shell/launch acknowledgement is lost, inspect once before retrying; never launch a second owner blindly. Confirm:
|
|
|
|
- one manager, one ownership lock per running session; no old mail deliverer, old orchestrator service or old session agent owners;
|
|
- cred starts **locked**, with no master-key/vault handover; stored grants/requests/settings remain intact; interrupted credential request callers retry their same stable key, not a secret-backed external mutation;
|
|
- only explicit target run-start-without-run-end work is recovered under its existing agent-run UUID; finished-looking old history alone is not resumed;
|
|
- pending new mailbox work and durable events catch up, callbacks tolerate repeats, fresh default/advisor POST retries use the same ID, no old pending queue/undelivered mail is replayed;
|
|
- scheduled jobs/grants/non-target state match the inventory except recorded intended schema/activity additions; retained `unknown` mail/tool outcomes remain visible;
|
|
- effective grace is 5 seconds regardless of legacy metadata, and crash supervision has not disabled a service. Deliberate `core.stop`/`core.kill` disables that service registration; do not use it as a crash-supervision test;
|
|
- first-update cuts are surfaced: old runs without target records cannot be recovered automatically; tools may already have external effects; matching histories do not prove an effect did or did not happen.
|
|
|
|
Repeat live-size post-record publication/catch-up measurements if production timing is a release gate; compare `mail.send` to the existing 3-second orchestrator / 10-second cred limits. Fast manager boot acknowledgement is not a completed initial scan. Do not turn catch-up into a timer/poll loop.
|
|
|
|
## 5. Release web, reopen ingress, delete the one-off executable
|
|
|
|
Only after session and web plugins are active and verified, mediator/orchestrator uses the existing **`orchestrator.release` workflow for SAU-375** to clear its `publish:hold` and publish its approved commit. Check that method's current typed arguments before execution; this draft intentionally does not invent them. No worker performs this action.
|
|
|
|
Verify the published approved asset hashes and ingress once, mark the migration verified, then **delete its executable and any executable cached/controller copies**. The rehearsal did this; live mediator must too. Retain only the manifest, backup/inventory/loss/caller-conversion records and logs as migration evidence, plus separately reviewed target artifacts. Do not retain an enabled migration plugin or boot entry.
|
|
|
|
## Failure/rollback boundaries
|
|
|
|
| Boundary | Action |
|
|
|---|---|
|
|
| Before offline mutation | Stop with evidence; old frozen world is untouched. |
|
|
| During swap, target never started | Resume the saved hash-guarded plan, or restore the full frozen backup and old external owner/configuration coherently. Do not partially start mixed contracts. |
|
|
| Any target start attempted | No automatic full restore/reapply. New histories, mail, jobs, grants or external effects may already exist. Preserve new state; diagnose and plan a narrow forward repair/revert. Never blindly rerun an external effect or undo new work. |
|
|
| Session revert after target activity | A new approved operation: close gateway/ingress, `session.unsubscribe {id:"web-chat"}` while the new manager is still available, verify removal, prevent gateway re-registration while old session runs, then coordinate the stopped version change. Do not restore a registry backup that reintroduces the registration. ID-only rename is proven unsafe. Run 1369 verified this with the actual gateway left running and no re-registration: zero historical callbacks after unsubscribe; ID-only re-key replayed history. Physical browser/SSE display is not claimed. |
|
|
|
|
Independent plugin recovery follow-ups remain independent. Neither uniform grace nor this one-off migration proves their external side effects, usage accounting, crash recovery or provider cancellation are solved.
|