Complete native owner/package cut-over rehearsal and operator manifest

Card: SAU-384
This commit is contained in:
SAU-384 implementer
2026-10-31 13:49:57 +00:00
parent 17d1c52eca
commit e4c8fcafbb
3 changed files with 104 additions and 20 deletions
+20 -11
View File
@@ -1,16 +1,18 @@
# One-off handover operator runbook — blocked draft
# One-off handover operator runbook — rehearsed, approval required
Read [rehearsal/results](SAU-384-rehearsal.md), [resume verification](SAU-384-resume-1369.md) and [inspection prerequisite](SAU-384-resume-1372.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes.
Read [rehearsal/results](implementation.md), [race/rollback verification](resume-1369.md), [native owner/package rehearsal](resume-1373.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes.
## 0. Finish the installation manifest (no live mutation)
## 0. Verify and freeze the installation manifest (read-only first)
Evidence root: `/docs/symphony/artifacts/SAU-384/evidence/`.
Bundle: `/docs/symphony/artifacts/SAU-384/target-artifacts.tar.gz`.
Selected coherent bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`.
The older `target-artifacts.tar.gz` remains prior rehearsal evidence, **not** the native installation selection.
- Verify bundle hash against `bundle-sha256.txt`, then every extracted target file against `manifest.json`. This bundle contains target binaries/static assets and the one-file SDK patch, **not** a migration executable.
- Record the actual native service owner/launcher, its synchronous stop/start commands, automatic host restart mechanism, runtime closure and native architecture. Manifest fields are deliberately null until inspected. The bundled host is a local **aarch64-linux debug-profile, stripped** build of `e09c4f9f`; it has not been established as the right live-host installation package. Stop here on a mismatch; build/rehearse the right artifact rather than substituting an untested binary.
- Verify the new bundle hash against `evidence/cutover-targets-sha256.txt`, then every extracted file against `evidence/manifest.json`. It contains reviewed plugins/static assets, SDK `src/lib.rs`, and `native-package.nar.gz` with its closure manifest, **not** a migration executable. It excludes the old diagnostic debug host.
- Native selection is the Nix release package `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux, built from this card's tree with unchanged host source `e09c4f9f`. `evidence/native-host-manifest.json` pins its NAR/file hashes and every runtime closure path/hash. All dependency paths match the observed old package's closure; only the host root differs. Recheck closure availability/hashes before execution. Mediator may import the exact reviewed NAR if needed (`gzip -dc native-package.nar.gz | nix-store --import`); never substitute the debug binary. Import/deployment is not a worker action.
- Owner is **Herdr workspace w7, pane w7:p6 (tab w7:t5)** running `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`, not a systemd Sausage unit. Re-identify its current loop/host PIDs using read-only `herdr pane process-info --pane w7:p6`; observed PIDs in `resume-1373-inspection.json` are evidence, never reusable constants. The loop refreshes a remote flake after any nonzero exit, including core.restart (75). Its existing 2-second restart sleep is not a migration quiescence mechanism.
- Pin all old plugin hashes, full boot bytes, current SDK source hashes, caller configuration, and `/proc/*/info.json` inventory. Record active agent/tool processes and any external effects whose result is uncertain. Do not infer settlement from the last assistant text.
- Inventory executable caller payloads in cron jobs, boot and other saved callback/action/configuration stores using their predecessor schemas. History text mentioning an old RPC is not an executable payload. The tested converter covers `jobs.data` where `method == session.send`; it does not claim a full live caller-store audit. Run 1372 observed six historical legacy session cron jobs (all jobs have `next_run:null`) and one already-target-valid board session hook. See `evidence/resume-1372-caller-inventory.json`; the frozen audit must also cover board `invariant_target.data`, hook attempts/subscription batches, room/orchestrator workflow records and any session event registrations. Diagnostic call snapshots and RPC names inside prompt strings must not be rewritten/replayed as executable configuration. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
- Inventory executable caller payloads using predecessor schemas, not substring rewriting. Run 1373 read-only audit covered cron jobs; board hooks, invariant target, attempt snapshots, subscriptions/batches; room/workflow events; orchestrator runs, skill reviews and host jobs; manager registrations and configuration. `evidence/resume-1373-inspection.json` records six legacy cron templates requiring conversion, an already-target-valid board hook **and invariant target**, eight historical running hook snapshots that must NOT replay, zero subscription batches, one abandoned room with no pending work, and no live session manager registrations. The only observed conversion beyond boot is cron `jobs.data` where `method == session.send`. Repeat against the frozen bytes: this observation is not a transactional final snapshot. Prompt strings/diagnostics are not executable templates; preserve them verbatim. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
- Source references for predecessor formats: session `f9122585`, `src/main.rs` Paths (`pending.jsonl`, `cancel`) and `src/queue.rs`; cron `ba75c102`; target cron README's one-off conversion procedure. Active old component hashes are separately captured; do not equate a source reference with its compiled active binary without checking.
- Confirm the frontend artifact is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`; keep `publish:hold` until step 5. Do not publish a worktree snapshot or release it early.
- Rollback uses the card's explicit **unsubscribe** option. ID-only re-key is **unsafe**, proven again with the actual old manager and gateway in run 1369. Prevent the gateway from re-registering `web-chat` against an old session binary; never call `web.start` in that state. A different proposed “re-key” needs its own defined semantics/rehearsal.
@@ -21,9 +23,9 @@ Surma's go must precede the live core/session/cred operation. Every push require
Run the one-off controller **outside the world and outside the service/process group being stopped**. A plugin or agent in this world cannot be the controller: stop-all must stop it too. Recreate the narrowly reviewed temporary executable only for this approved operation, from the logged implementation with the resolved manifest; never add compatibility/transition code to a plugin.
Close the actual ingress and stop the **external host/service owner** synchronously using the pinned command. Await its shutdown/process-exit completion, not a loop over `/proc`, endpoint retry, sleep, or a set of stop requests. If an outer launcher would respawn the host, deliberately stop that owner too; do not edit boot config while agents can still run. `core.restart`/exit-75 is a restart request, not stop-all. The real core's `Host.stop` suppresses service restart admission and stops all trees in parallel. Design budgets are grace 5 seconds and separate EOF-stop 5 seconds, not arbitrary new timeouts.
Close ingress before stopping ownership. The tested exact native boundary is the deleted one-off `native_freeze.py` (source-as-log linked in resume-1373): invoke its reconstructed, reviewed copy **outside the world/group**, with current loop PID, current host PID and an external checkpoint path. Use external root `/home/surma/sau-384-cutover`, with the reconstructed scripts in `tool/` and checkpoint `evidence/freeze.json`. The observed Python interpreter is `/nix/store/33i3kijbra0cffncn3rmi9bdrav7a3y3-python3-3.13.15-env/bin/python3`. After Surma's go and fresh PID/identity inventory, the exact stop invocation is `PYTHONDONTWRITEBYTECODE=1 <that-python> tool/native_freeze.py "$LOOP_PID" "$HOST_PID" evidence/freeze.json` from the external root. Do not pass the rehearsal-only failure-injection argument. It pins boot ID/start ticks/executable/group and pidfds, sends SIGTERM to the **loop first**, awaits loop exit, checkpoints `owner_stopped`, then sends SIGTERM to the exact host and awaits its exit. Host SIGTERM awaits real `Host.stop`; its old source was inspected at the actual Nix derivation. After both exits, inspect the saved group once; any unexpected survivor blocks without blindly killing it. No process polling, endpoint retry, sleep or service-stop-request counting. Do not close the Herdr server or unrelated panes.
The rehearsal's native external controller awaited `Host.stop`, asserted its process table was empty, then awaited host exit. The live equivalent needs the actual launcher identity from step 0. **That identity is currently missing; do not invent a unit name.** Source config has `services.sausage.enable = false` and describes a Herdr panel; read-only systemd-web inspection corroborates a user Herdr socket, not a system `sausage.service`. Credential request `055995a51608c57dd25b323bea731fcb` is approved for read-only inspection, but SSH refused the unknown Citadel host before authentication. Mediator must install the observed public host-key line from `evidence/resume-1372-ssh-trust.json` in `/config/known_hosts` and resume inspection; worker configuration changes remain forbidden. No additional person approval is needed for that trust entry under the SSH skill. It is not permission to stop or change the launcher.
The non-parent-controller rehearsal proves loop exit does not kill the migration controller or old host prematurely; the host survives its parent and drains on its own signal. Interrupting after loop exit is resumable from the same pinned checkpoint; stale PID identities refuse. Even a real core.restart after loop exit cannot respawn the host. `core.restart` by itself is **not** stop-all. Core's shutdown suppresses service restart admission and stops all trees in parallel. Design budgets remain grace 5 seconds and separate EOF-stop 5 seconds; the controller awaits completion rather than inventing a live timeout. Never edit world state before the `frozen` checkpoint.
## 2. Back up frozen state and plan changes
@@ -35,7 +37,7 @@ Only after shutdown completion:
4. Convert each old cron `session.send` template once: require a JSON object/nonempty text; refuse `interrupt:true`; remove false/absent legacy `interrupt`, inner `detach`, `notify`, permanent `id`; set absent kind to `follow-up`; preserve all other template metadata/settings verbatim; set outer detach false. Preserve SQL IDs and every schedule/last-run/outcome field. Update all affected jobs in one `BEGIN IMMEDIATE` transaction, validate readback/integrity, and preserve non-session job JSON bytes. Record removed notify targets for review—do not replace them with an invented answer-forwarder.
5. Stage target host and all affected plugin components from the resolved manifest as one offline cut. Do not call live `core.activate` for piecemeal contract upgrades. Unaffected plugin state/binaries stay intact. Copy only SDK `src/lib.rs` for this selected baseline; preserve relocated world Cargo manifests, imports and `interface`. If hashes differ from this baseline, resolve the coupled source-copy set rather than applying this one-file assumption blindly.
The native host/runtime installation itself remains gated on the unresolved owner/architecture/closure in step 0. The rehearsal exercised real host shutdown plus offline plugin/SDK/boot replacement, not replacement of the live native service package.
Native host installation is an **outer launcher selection**, never an overwrite of immutable `/nix/store`. While stopped, install the static selector flake recorded in the manifest at `/home/surma/sau-384-cutover/target-selector/flake.nix` (outside the world). The proposed parent was absent at read-only inspection; refuse an unexpected existing image rather than overwrite it. Its sole app points to the exact target package's `bin/sausage`. Hash-guard and back up any existing selector; do not overwrite an unknown image. The selector is target launcher configuration, not a migration executable. The actual old Nix package/loop → offline plugin/SDK/boot swap → new Nix package/loop was rehearsed; neither public remote refresh nor a source-only host was substituted.
## 3. Install boot/caller configuration while stopped
@@ -57,7 +59,14 @@ Resume an interrupted **offline** swap only from its saved plan: each file must
## 4. Start/reconcile and verify
Start the pinned target external owner once. Observe readiness/completion events, not process/endpoint polling. Confirm:
Checkpoint `started` **before any launch attempt**. In the identified Herdr shell, mediator starts the same inspected loop exactly once with an explicit pinned selector:
```sh
SAUSAGE_FLAKE=path:/home/surma/sau-384-cutover/target-selector \
/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/
```
The selector directory must contain exactly the recorded installation bytes from step 2, not a mutable public flake. This pins crash restarts to the same reviewed host instead of refreshing unknown future code. Keep that selector/closure reachable for the target owner after deleting the migration tool. Observe boot/RPC/watch completion events, not process/endpoint polling. If shell/launch acknowledgement is lost, inspect once before retrying; never launch a second owner blindly. Confirm:
- one manager, one ownership lock per running session; no old mail deliverer, old orchestrator service or old session agent owners;
- cred starts **locked**, with no master-key/vault handover; stored grants/requests/settings remain intact; interrupted credential request callers retry their same stable key, not a secret-backed external mutation;