Close cut-over recovery, executable-caller, arm and deferred Git gaps
Card: SAU-384
This commit is contained in:
SAU-384 implementer
@@ -6,11 +6,13 @@ Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover
|
||||
|
||||
## Closed integration gaps
|
||||
|
||||
- **Round 3, run 1389:** deferred approved git 9cc9fa6f is bundled/preflighted, activated only after target cred is live and Surma unlocks. Real old/new Git + new cred loopback tests covered granted SSH, locked denial and new shared SSH. Added real-driver pre-start abort/restore/old-owner relaunch, live consistent-copy/executable-only caller guards, special-file/free-space gates, finite registration deadline/failure notification, exact pre-cut prev images and durable arm-marker crash recovery. Real partial-swap/ENOSPC/audit failures, arm interruption, final Herdr cutover/preservation/sidecars and full workspace (208 host tests) passed. Tool/controllers deleted after verification. Exact logs, selection and limits: [rework-1389.md](rework-1389.md); proposed live recovery/roots-side-effect plan: [runbook](runbook.md).
|
||||
|
||||
- **Review rework, run 1381:** one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See [rework-1381.md](rework-1381.md) for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). [Runbook](runbook.md) now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.
|
||||
- **Run 1373:** after mediator installed the observed public host key, approved read-only SSH inspection identified the actual Herdr pane, outer `sausage-loop`, world path, native architecture, old package/derivation/source and runtime closure. The actual old Nix package/loop → stop-all → offline swap → exact new Nix release package/loop passed in the disposable live-size world. A non-parent external controller separately proved loop-first stop, interruption/resumption, stale-PID refusal and safe frozen rerun. A real core.restart after loop exit could not respawn the host. See [native evidence](resume-1373.md).
|
||||
- **Run 1369:** armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed. Hooks existed only in a disposable host copy, never the selected host artifact or committed code.
|
||||
- **Rollback:** actual old-manager/gateway tests prove **unsubscribe works; ID-only re-key does not**. The old binary ignores best_effort and replays history to a renamed ID. The runbook mandates the card-authorized unsubscribe option and prevents web re-registration during a session revert. A physical browser/SSE display is not claimed.
|
||||
- Final selected coherent bundle is **`cutover-targets-1381.tar.gz`** (SHA-256 `9295c5fddbf5d68cbfd525602ec67e7dd895bf6afb99324191a6c514282eac57`) with the same Nix native package/closure, approved cred main `0db40101` rebuilt as **1636 / 274 tests**, other reviewed plugins/web assets and SDK `src/lib.rs`; it excludes the diagnostic debug host. The required last main merge found SAU-418, so the affected native/sidecar/preservation/full-workspace gate was repeated and the temporary tool deleted again; see rework-1381.md, Final main refresh. The older `target-artifacts.tar.gz` remains prior evidence. Executed one-off tool/controllers were again deleted after verified success; source survives as log only.
|
||||
- Final selected coherent bundle is **`cutover-targets-1389.tar.gz`** (SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`), **23 payloads**, including deferred approved **git 9cc9fa6f / 177 tests**. Same Nix native package/closure, approved cred main `0db40101` build **1636 / 274 tests**, other reviewed plugins/web assets and SDK `src/lib.rs`; no diagnostic debug host. Git is not swapped offline; Surma unlock precedes its mediator activation. Earlier bundles and runs remain historical evidence. Final exact-driver native/preservation/sidecar/failure and full-workspace gates passed again; executable/controller deletion and source-as-log are in rework-1389.md.
|
||||
|
||||
The final frozen inventory/caller-schema/hash check remains an operator gate at the actual approved shutdown, not something a read-only rehearsal can perform in advance. Live execution still requires Surma's go and mediator/orchestrator action; no approval request is pending.
|
||||
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# SAU-384 round 3 — run 1389
|
||||
|
||||
Ready for review; **not authorization for live execution**. Addresses `/docs/symphony/artifacts/SAU-384/extra-round-3.md` and review 1388. Only documentation and disposable tooling changed; no transition code in any shipped plugin/host.
|
||||
|
||||
## Reviewed selection and proposed plan
|
||||
|
||||
- Selected bundle: `cutover-targets-1389.tar.gz`, **27,271,968 bytes**, SHA-256 **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**. Embedded manifest SHA-256 **b48f2dacda3f348a67574767c6accfae1f1102227f94f86868ec1436bd4df394**; all **23** payload sizes/hashes checked. Prior bundles are immutable historical evidence.
|
||||
- New deferred payload: approved git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests, not rebuilt here). Staged world copy: `/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm`. Both manifest `deferred_plugins` and operator `deferred_artifacts` are hash-checked before freeze. **Not installed offline**: mediator activates it only after target cred is live and Surma unlocks through the UI. The runbook uses the real `core.activate` **wasm** argument and includes the activation check before web release/final deletion.
|
||||
- Cred remains approved **0db40101**, build **1636 / 274 tests**. Mail/cron/rooms remain **1600/1602/1603**. No new plugin build/source edit this run. Native host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, source e09c4f9f; old host mggvibk8 and loop pz2b94a2 remain pinned. SDK-copy list: **src/lib.rs only**. Web remains approved **3d760267**, release held until session/web/verification prerequisites.
|
||||
- Exact external commands, operator/approval boundaries, checkpoint archive/retry and rollback rules: [runbook.md](runbook.md). Live manifest template: `evidence/round3-1389/live-manifest.template.json`. Resolve fresh identities/caller guards before go; sandbox manifests are evidence, not live instructions.
|
||||
- **Expected live Nix side effect requiring inclusion in Surma's go:** three `nix-store --query --roots` calls, for old host, target host and loop, may prune stale auto-root and temporary-root registrations. They do not delete store packages. Five root registrations (those three plus controller/Herdr) remain in the proposed operation. This round used only the **private rehearsal Nix DB**; no shared/live roots query was repeated. Prior incident remains disclosed in rework-1381.md.
|
||||
|
||||
## Six findings fixed
|
||||
|
||||
1. **Git and interim compatibility.** Actual live old WASM (reported source f089b56d; SHA-256 4b460bce9b67552e223bb209591009056d409b77290182f6cf142adc8c4c97f4) ran against real new cred in a separate synthetic world. Explicit-grant SSH clone authenticated with a real loopback sshd, recorded exactly one sign use and fetched the synthetic commit. Locked use returned `vault_locked`. Old grant-free shared SSH still fails requesting a local key; old grant-free HTTPS retains its old config/anonymous behavior, not the new sharing switch (source checked). Old granted HTTPS reached/was accepted by new cred and reserved one use, then the deliberately closed loopback destination returned `network_error` plus capture; **successful HTTPS transfer is not claimed**. After actual synthetic unlock and new-git activation, shared grant-free SSH clone succeeded. Do not schedule Git network work in the locked/old-git interval, introduce key/config-secret workarounds, or blindly retry an already-started caller's errors. Local status/commit/main merges are unaffected. Live secret-backed operations/pushes remain untested and separately approval-gated.
|
||||
2. **Pre-start abort and human operator.** `check`/start, arm and immediate pre-freeze gates inspect consistent live SQLite backup-API copies, special file types, caller/file/artifact guards and free space. Space budget derives from current regular-file logical bytes plus the actual staged plugin/prev/SDK and recovery/stage/audit DB copies; no arbitrary multiplier/headroom/default. Growth or later ENOSPC is still possible. Driver now has a real `abort` CLI: exclusively lock, finish freeze if begun, restore only approved before/after paths from the complete immutable backup, verify full frozen inventory, then checkpoint-before-launch the explicit old owner in the same pane. No completed plan means apply never began: do not restore an incomplete backup. Restore is resumable; after `old_start_attempted` inspect instead of launching twice. Surma is the named external operator, with independent terminal log monitoring, `failure.json`, and best-effort Herdr notification (submission result retained; physical display not claimed). Runbook gives exact resume/abort and evidence-archive/re-resolution commands. After target `started`, automatic resume/restore/relaunch stays forbidden. Missing registration now exits on one absolute **120 s** deadline from cron/src/types.rs's existing registration/publication budget and the earlier real-core fixture bound; Herdr/RPC calls are finite too. No polling for quiescence or availability.
|
||||
3. **Executable-only SQL guards.** Cron projects id/method/params_json/cron/at/timezone/detach/call_timeout_secs, not next/last/result/occurrence bookkeeping. Board excludes last_sent/ended and batch IDs/lines/due; unions active subscription config with pending config snapshots so creating/delivering the same active configuration is not drift. New/changed executable configuration still refuses. Final automatic subscription removal changes the executable set and requires refresh or pre-start abort, explicitly documented. Add any planned mediator cron wake-up **before** resolving/approving operator.json. No caller/boot/plugin/SDK edits between resolution and freeze; arm checks again. No in-progress manifest edits.
|
||||
4. **prev contract.** Every offline-swapped plugin's prev is now its exact **frozen pre-cut current**, under before/after guards and restore. Tests checked all nine, including paths where an older prev existed. Git waits for normal later activation/prev behavior. This makes the fallback image meaningful, **not** a license for blind incompatible individual `core.revert`; coordinated rollback and unsubscribe-before-session-revert remain mandatory.
|
||||
5. **Roots side effect.** Listed above and in the live runbook/approval plan, rather than describing roots queries as mutation-free. Subsequent tests stayed private.
|
||||
6. **Arm race.** Replaced reader-dependent FIFO with atomic, manifest-bound **arm.json** and an arm-then-scan inotify watch. Idempotent arm returns with no reader. `resume` accepts waiting_for_arm; a kill after marker observation but before saving armed reuses the same marker. No manual FIFO writer/cleanup is needed.
|
||||
|
||||
## Executed gates and full logs
|
||||
|
||||
Prefix: `/logs/box/2026-10-31/tc-asa3yr-exec-`.
|
||||
|
||||
| Gate | Log / outcome |
|
||||
|---|---|
|
||||
| Final exact tool, caller killed + controller killed/resumed, incomplete backup resume, real Herdr job-control stop/swap/start, same shell, post-start preservation/new verifier, prev hashes, pinned cold restart and old clone | **3699755631.stderr**, pass. Final backup 4,436 files / 627,157,113 bytes, 4.559 s (session-size integration, not full-world downtime). No old mail owner; one manager; cred locked. |
|
||||
| Live special-file/zero-space negative, mutable cron vs changed payload, pre-arm drift, frozen SQL mismatch -> real-driver abort/old relaunch, consumed-arm/save kill/resume, post-start abort refusal | **1485773773.stderr**, pass. Zero-space gate used a test-only mocked disk-usage response; actual ENOSPC tested below. |
|
||||
| Real-driver partial swap -> abort/restore/old relaunch; actual backup ENOSPC on a private tmpfs introduced after good preflight -> untouched-old relaunch; lost console subscription -> bounded clean exit | **4142682240.stderr**, pass. Registration wait 120 s; elapsed 132.493 s including backup/launch. Started checkpoint refused both resume and abort. |
|
||||
| Real old/new Git + new cred, loopback synthetic OpenSSH/HTTP negative, unlock then deferred activation | **99781693.stderr**, pass. No live key/grant/remote used. |
|
||||
| SQLite WAL + genuinely hot rollback journal, exact losses, preserved job IDs/calendar/outcomes/generic JSON/uncertain mail, apply interruption/rerun and exact physical restore | **1727732043.stderr**, pass. |
|
||||
| Board executable projections and old-select-star negative control; template/deferred-artifact checks; final driver's actual Herdr failure notification + abort | **1276236846.stderr**, pass. |
|
||||
| `cargo test --workspace` against unchanged production source | **2259546796.stderr**, pass: **208 host integration tests, lines 62–431**, plus all workspace suites/doc tests. 133 Rust/Cargo/flake inputs matched card tree. No Rust implementation change. |
|
||||
| Executed verify mode, complete source-as-log and deletion | **2013747101.stderr**, pass. See hashes/ranges below. |
|
||||
|
||||
Prior accepted race/replay/deadline/core-watch/upgrade/idle/grace/crash-supervision evidence is cumulative, not silently rerun or claimed fixed here. Prior live-size scale sample remains **138,444 files / 12,441,162,164 bytes / 45,310 directories**, synthetic size-bucket match, 205.940 s inventory+copy/hash, warm cache. Post-record manager ack 0.6078 s, registration catch-up 0.0128 s, mail publication 0.5376 s (integrated 0.5235 s) remain under caller publication budgets **3 s orchestrator / 10 s cred**; catch-up outlier 12.0419 s and initial-verifier startup scan delay 23.753/24.530 s remain disclosed. No production downtime/latency guarantee.
|
||||
|
||||
## Deleted executable and reconstruction receipt
|
||||
|
||||
Only source-as-log remains in **2013747101.stderr**. Reconstruct exactly three files for the later approved operation, hash-check before use; do not reconstruct test controllers:
|
||||
|
||||
- cutover_once.py: lines **2–164**, SHA-256 **3adf65ebb0b3401a0d1976524f35f7450e907ad32f907dcbb1cd2f7e0a7e5d3a**;
|
||||
- freeze.py: lines **166–238**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
|
||||
- driver.py: lines **240–503**, SHA-256 **686167318c84a32c1010daf4517f036bcb654384cadab987fa47fb13deecd2bf**.
|
||||
|
||||
`verify` actually logged/deleted the executable after successful rehearsal. Executed test/patch/controllers and test launchers were logged/deleted too, with bytecode caches removed. `tool-deletion.json` and `executed-test-deletion.json` are under `evidence/round3-1389/`. Permanent pinned owner/selector configuration is not migration code; it stays rooted through the recovery window. The live runbook still requires mediator deletion after verified success.
|
||||
|
||||
## Failures and limits
|
||||
|
||||
Tried next methods, not blockers: initial generated SQL fixture had quoting syntax wrong (corrected); generated lockfiles made whole-directory diff reject (compared actual Rust/Cargo/flake input bytes, then ran full suite); final-sidecar extraction included a trailing log status (trimmed); a post-check initially read the previous round's stopped-world manifest (corrected to this round, then passed). Synthetic SSH setup needed explicit typed key metadata, a sandbox-only sshd account and an unlocked sandbox login account; default root was locked. A failed fixture left two loopback sshd listeners, later explicitly identified/stopped. Initial sandbox activation used `path`; core required `wasm`, corrected and the complete final Git gate passed. No failed test was relabeled successful; full errors remain in logs.
|
||||
|
||||
Still unverified/out of scope: approved live cut-over/frozen caller snapshot, actual live SSH-detach/pane w7:p6 execution, Surma's real unlock/new-git live networking, physical Herdr notification/browser display, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. No live activation/restart/config/owner change, push, release, deploy or /www publication this run. Surma's go is still required before mediator core/session/cred operation, detached SSH exception, roots side effects and **every push**.
|
||||
+66
-19
@@ -1,6 +1,6 @@
|
||||
# One-off handover runbook — rehearsed; live approval required
|
||||
|
||||
Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
|
||||
Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md), [round-3 recovery/git gates](rework-1389.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
|
||||
|
||||
## 0. Prepare everything while the old world is still available
|
||||
|
||||
@@ -8,7 +8,7 @@ Do not begin downtime with scripts, caller decisions or launch instructions stil
|
||||
|
||||
### Reviewed selection
|
||||
|
||||
- Bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets-1381.tar.gz`, SHA-256 `9295c5fddbf5d68cbfd525602ec67e7dd895bf6afb99324191a6c514282eac57`. It includes final-main cred `0db40101`, build **1636 / 274 tests**, preserving approved SAU-418 instead of downgrading it. The earlier `cutover-targets.tar.gz` remains immutable prior evidence, not this final selection. Verify all 22 payloads against its embedded `manifest.json`. `target-artifacts.tar.gz` is older debug-host evidence, **not** this installation.
|
||||
- Bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets-1389.tar.gz`, SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`. It includes final-main cred `0db40101`, build **1636 / 274 tests**, preserving approved SAU-418 instead of downgrading it. Git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, approved SAU-418 artifact `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests), is included as **deferred_plugins/deferred_artifacts**, not an offline replacement. It activates only after target cred is live AND Surma unlocks it in the UI. Earlier bundles are immutable historical evidence, not this selection. Verify all **23** payloads (including deferred git) against its embedded `manifest.json`. `target-artifacts.tar.gz` is older debug-host evidence, **not** this installation.
|
||||
- Native host: `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux Nix release package from unchanged host source `e09c4f9f`. The embedded `native-host-manifest.json` pins the NAR, binary and complete runtime closure. Import the reviewed NAR if needed; never overwrite immutable store files or substitute a diagnostic host. Import/deployment is an operator action, never a worker action.
|
||||
- Old package for this observed baseline: `/nix/store/mggvibk8lcgv2qw28jimrl60mi0qm41p-sausage-0.1.0`. Re-identify it and the owner before go; observed PIDs are not constants.
|
||||
- Actual owner: Herdr **w7:p6**, tab w7:t5, not a systemd Sausage unit. Its loop is `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`. Read `herdr pane process-info --pane w7:p6` once to resolve current shell/loop/host identities. Do not close Herdr or unrelated panes.
|
||||
@@ -17,14 +17,14 @@ Do not begin downtime with scripts, caller decisions or launch instructions stil
|
||||
|
||||
### External driver and resolved operator manifest
|
||||
|
||||
Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-asa3yr-exec-3977808892.stderr`; check their hashes against `evidence/rework-1381/tool-deletion.json`. These are the executed/reviewed versions, not the earlier scripts from run 1373. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
|
||||
Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-asa3yr-exec-2013747101.stderr`; check their hashes against `evidence/round3-1389/tool-deletion.json`. These are the executed/reviewed round-3 versions, not the earlier scripts from runs 1373/1381. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
|
||||
|
||||
The tested interpreter is `/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3`, with **inotify-simple 2.0.1**. Resolve its closure availability before go. After go and before starting the driver, pin its package and the inspected Herdr package with host-visible GC roots for the operation (`nix-store --realise <exact-package> --add-root <external-root>/gc-controller --indirect`, and equivalently `gc-herdr`). These root writes/imports are included in the approved live plan, not preparation actions authorized to a worker. The driver itself registers and verifies **old host, target host and loop** roots at `<root>/gc-{old,target,loop}` before freeze. Use the **host namespace**, not a box talking to shared Nix state: `/rework` roots in a box are not host-visible. Do not run GC as a test. Keep old/target/loop roots and both selectors for recovery after tool deletion.
|
||||
|
||||
Start from `evidence/rework-1381/main-refresh/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
|
||||
Start from `evidence/round3-1389/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
|
||||
|
||||
- native paths, Herdr binary/context, pane and current shell/loop/host PIDs;
|
||||
- external root, world, bundle, embedded manifest SHA, artifact paths, SDK path, exact target boot list;
|
||||
- external root, world, bundle, embedded manifest SHA, offline artifact paths **and deferred git**, SDK path, exact target boot list; root and world are on the same filesystem in this plan; resolve a different storage topology before go;
|
||||
- complete old in-flight `/proc/*/info.json` and native-owner inventory, plus uncertain external effects; a finished-looking assistant tail is not settlement;
|
||||
- a new continuation UUID, generated **once**, and the explicit new mediator verification instruction. Keep that UUID on all retries;
|
||||
- typed executable-caller review and its frozen guards below.
|
||||
@@ -35,13 +35,13 @@ Do not copy sandbox PIDs, `gc_store`, or `test_gate_after_freeze` into a live ma
|
||||
|
||||
Use predecessor formats: session source `f9122585` Paths/queue; cron `ba75c102` and current cron README one-off procedure. Do not substring-rewrite prompts or diagnostics. Review current stores as in `resume-1373.md`, then encode the **approved** file hashes and deterministic SQL projections in `caller_file_guards` and `caller_sql_guards`:
|
||||
|
||||
- cron `state/cron/jobs.sqlite`: `select id,data from jobs order by id`;
|
||||
- board `state/board/board.sqlite`: deterministic projections of `hooks`, `invariant_target`, `subscriptions`, and `subscription_batches`, using the actual current schema. Review hook/invariant templates. Historical hook-attempt snapshots remain diagnostics and **must not replay**;
|
||||
- cron `state/cron/jobs.sqlite`: use the template's ordered `id, json_extract(data,...)` projection of **method, params_json, cron, at, timezone, detach, call_timeout_secs**. Do not hash next_run, last_run, last_result_json or last_occurrence_id;
|
||||
- board `state/board/board.sqlite`: ordered `id,data` for hooks/invariant_target; `id,target,session,data` for subscriptions; and `select id,data from subscriptions union select subscription,data from subscription_batches order by 1,2` for executable subscription snapshots. Ignore last_sent/ended, transient batch IDs/lines/due; a pending delivery of the same active configuration does not change this union. Historical hook-attempt snapshots remain diagnostics and **must not replay**. Automatic final subscription removal changes the executable set; resolve/refresh before arm, or take the pre-start abort path if it happens after arm;
|
||||
- boot and other discovered executable configuration/manager registrations. Review typed room/workflow/orchestrator stores too; preserve their histories/settings and non-executable records. Do not mistake prompt text for an RPC template.
|
||||
|
||||
A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
||||
A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. These projections, including all column names, live in the round-3 template and were tested against mutable cron/board bookkeeping. Any genuinely new executable store/field still needs review; do not silently omit it. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
||||
|
||||
After freeze the driver backs up all bytes, checks these projections on **disposable complete DB copies**, and verifies file guards against the frozen backup **before apply**. It never recovers/opens immutable backup evidence in SQLite. A changed projection stops offline with evidence; there is no in-world human gate after stop-all. Include any newly discovered executable store in the approved manifest before arm.
|
||||
**Before start, again before arm, and once immediately before freeze**, `check`/driver checks artifact and file guards, SQL projections on consistent live SQLite backup-API copies, inventory special-file types (without hashing mutable bytes), and available space. The space budget is the measured current regular-file logical bytes plus staged plugin/prev/SDK bytes and the concrete recovery/stage/audit DB copies; the receipt saves required/free bytes. Root/world free space is checked before downtime; further log growth, changed mounts or disk exhaustion can still occur. After freeze the driver backs up all bytes, checks these projections on **disposable complete DB copies**, and verifies file guards against the frozen backup **before apply**. It never recovers/opens immutable backup evidence in SQLite. A changed frozen projection stops before apply with `failure.json`, log and a best-effort Herdr notification. Surma uses the exact pre-start abort command below; there is no in-world human gate after stop-all. Include any newly discovered executable store in the approved manifest before arm. Add the mediator's planned cron wake-up **before resolving/hash-approving operator.json**, never between resolution and freeze. From resolution until freeze: no job add/remove/template/schedule edits, hooks/subscription/invariant/manager-registration changes, boot/SDK/plugin activation or other executable-caller changes. Routine next/last/result and subscription delivery bookkeeping is permitted. If new intent is needed, cancel unarmed or abort pre-start, re-resolve, and obtain approval for the changed manifest; do not edit a hash-bound in-progress manifest.
|
||||
|
||||
### Approval and external launch surface
|
||||
|
||||
@@ -49,20 +49,25 @@ Surma's go must explicitly cover this resolved manifest, the stop-all core/sessi
|
||||
|
||||
The proposed concrete execution surface is **two finite foreground `ssh.exec` calls as surma on citadel.vpn.surma.technology**, using the existing approved signing grant, with commands `cd /home/surma/sau-384-cutover && <the start command below>` and then the same prefix with `<the arm command below>`. Each command terminates; the reviewed native CLI, not a shell `&`, creates the detached OS child. **Ask Surma's go to explicitly authorize this planned exception to the ssh-exec skill's usual prohibition on creating background work through exec strings. Until that explicit authorization, do not launch.** Do not improvise a tunnel, SSH PTY, new worker session or an in-world controller. This is a fully prepared proposed live operation, not a claim that the current worker has permission or has tested a live SSH launch. The same finite foreground surface executes the final verify/delete command after target verification. After arm, loss of the old mediator/SSH RPC result is expected and does not terminate the already-detached controller; inspect the external checkpoint instead of retrying start.
|
||||
|
||||
## 1. Start, acknowledge, then arm the external controller
|
||||
## 1. Surma monitors; mediator starts/arms the external controller
|
||||
|
||||
Close ingress and record accepted first-update cuts before arm. From the external root, on the approved host execution surface, mediator executes:
|
||||
Close ingress and record accepted first-update cuts before arm. **Surma is the external human recovery operator while the world is down.** Before arm he opens an independent Citadel terminal, changes to `/home/surma/sau-384-cutover`, and runs `tail --follow=descriptor driver.log` (GNU tail on this local filesystem uses inotify; no log rotation during the attempt) after start creates it. This terminal must not be an in-world session/tool process. The driver writes `failure.json` and emits `CUTOVER FAILED` on failure, and submits one best-effort Herdr notification `Cut-over needs Surma`; notification submission is saved in `failure-notification.json`, but no connected-client display is guaranteed. Terminal monitoring remains mandatory (SIGKILL/power loss cannot send a notification). Inspect checkpoint/controller identity once if the controller is unexpectedly gone, never repeatedly poll.
|
||||
|
||||
From the external root, on the approved host execution surface, mediator first runs the finite preflight command, then starts:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py check operator.json
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py start operator.json
|
||||
```
|
||||
|
||||
`start` launches a new OS session with stdin closed and output in `<root>/driver.log`, records its PID/start identity, returns an acknowledgement, and leaves the controller blocked on an **arm FIFO**. Check that receipt and checkpoint once, then:
|
||||
`start` launches a new OS session with stdin closed and output in `<root>/driver.log`, records its PID/start identity, returns an acknowledgement, and leaves the controller waiting on an inotify watch for **durable arm.json**. There is no arm FIFO or reader-dependent write. Check that receipt and checkpoint once, then:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py arm operator.json
|
||||
```
|
||||
|
||||
`arm` repeats live preflight and atomically writes a manifest-bound marker; re-arming is idempotent while waiting, including when no controller is alive. A crash after observing the marker but before saving `armed` resumes from `waiting_for_arm` and reads the same marker.
|
||||
|
||||
From here **the external driver, not the old mediator**, owns steps 2–4. The old default run and its tool calls are deliberately cut. They are not recovered from legacy history. The driver uses an external advisory lock to exclude another controller. Rehearsal killed both the invoking caller shim and, separately, the detached controller after freeze; checkpoint resume completed the chain.
|
||||
|
||||
## 2. Driver: freeze → full backup → plan/audit → offline apply
|
||||
@@ -73,7 +78,7 @@ The driver verifies reviewed bundle/plugin/SDK bytes and native closure hashes,
|
||||
2. Signals the exact host **SIGTERM then SIGCONT**, so a job stopped by background-PTY SIGTTIN can handle termination, and awaits host exit. Checks the saved group once for unexpected survivors; never blindly kills them. Saves `freeze.json` phase **`frozen`**, world path, boot ID and empty group inventory. No process/endpoint polling, sleeps or stop-request counting. Host shutdown stops all plugin/agent trees and suppresses restart admission. `core.restart` alone is not stop-all; after the loop is gone, exit 75 cannot respawn it.
|
||||
3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks **without following links**. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Unknown images refuse.
|
||||
4. Stages the approved transformations, then checks the frozen caller guards. **Both plan and apply independently require the actual frozen checkpoint**, same world/current boot, dead pinned owners, empty old group and no subsequently running host for this world. A caller-supplied `{remaining:0}` literal is not accepted. Changed checkpoint bytes also refuse.
|
||||
5. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. `migration/plan.json` reaches `installed` only after all operations complete.
|
||||
5. Stages **plugin.prev.wasm = that plugin's frozen pre-cut plugin.wasm** for every offline-swapped plugin, then target plugin.wasm. Both prev/current writes have before/after hashes and participate in restore. It never retains the unrelated older prev as the cut-over fallback. Git is deliberately untouched here; its later standard activation will save its own preceding binary. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. `migration/plan.json` reaches `installed` only after all operations complete.
|
||||
|
||||
Exact logical losses are **only** old session `pending.jsonl`, `cancel`, and unclaimed `mail.deliveries` rows with `state='pending'` and recipient not `surma`. Original messages, Surma copies, done rows and uncertain `unknown` deliveries stay; do not replay them. Loss records and original bytes/rows remain in backup/evidence. Histories/settings, scheduled SQL IDs/calendar/outcome fields, grants/requests and unrelated files remain intact.
|
||||
|
||||
@@ -105,17 +110,19 @@ The permanent `<root>/target-owner` exports `SAUSAGE_FLAKE=path:<root>/target-se
|
||||
|
||||
`core.restart` now restarts the pinned reviewed host; it does **not** self-update from gitea. Future host updates require a separately approved package/closure/root/selector change. A remote push alone must never silently choose an old host against target state. Keep both GC roots through the rollback window; retire the old root only after an explicit decision, not tool deletion.
|
||||
|
||||
**Expected approved live Nix side effect:** the driver makes three `nix-store --query --roots` calls, one each for old package, target package and loop, immediately after rooting those paths and before freeze. Those queries can prune stale auto-root registrations and stale temporary-root files in Nix state, even though they do not delete store packages. Surma's go must include that side effect (and the five root registrations including controller/Herdr); the earlier shared-store incident is disclosed in rework-1381.md. Rehearsal now uses only a private Nix DB; no live root query was rerun.
|
||||
|
||||
## 4. Driver: started → launch in the same pane → fresh verifier
|
||||
|
||||
The driver checks the pinned shell still owns the selected pane and no unexpected foreground process exists. It checkpoints **`started` before any launch attempt**, then submits the permanent target-owner path once via `herdr pane run` to the manifest's pane (live **w7:p6**). No shell reconstruction or in-world mediator action is needed during downtime.
|
||||
|
||||
Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. Then await actual gateway RPC completion (`cred.status` locked and registration acknowledgement). **Herdr wait-output is internally polling and is not used by the final driver.** It was an unsuccessful preliminary approach only.
|
||||
Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. The wait has **one absolute 120 s deadline**, from cron `src/types.rs` AddParams' existing registration/publication budget (also the earlier test_driver completion bound); it is not a periodic scan. A missing registration writes failure evidence, submits the Surma notification, and the controller exits. The `started` checkpoint remains: **do not resume, restore or launch again automatically**. Herdr/RPC calls also use that existing finite bound. Inspect the pane and native identity once, preserve new work, then arrange a narrow repair/new explicit verifier. `driver.py verify` is still allowed after human verification; no automatic retry occurs. Then await actual gateway RPC completion (`cred.status` locked and registration acknowledgement). **Herdr wait-output is internally polling and is not used by the final driver.** It was an unsuccessful preliminary approach only.
|
||||
|
||||
Finally the driver publishes the manifest's **new explicit default verification message** with its stable UUID and records `awaiting_verification`. The manager, not the driver, runs it. This names the new mediator as verifier without claiming the cut default run was recovered. The test produced one run-start and mailbox-consumed pair for that message. Publication is not immediate model execution: in the last gate it waited **23.753 s to run-start / 24.530 s to consumption** behind initial reconciliation. Do not confuse that with publication latency or a completed startup scan.
|
||||
|
||||
No automatic relaunch, whole-world restore or reapply is allowed after `started`. If launch/acknowledgement is lost, inspect the pane/checkpoint once before any repair. Do not launch a second owner blindly.
|
||||
|
||||
## 5. New mediator: verify → release web → delete tool
|
||||
## 5. New mediator: locked proof → Surma unlock → git → release web → delete tool
|
||||
|
||||
The explicit new verification run reads the manifest, `driver.json`, migration/loss/audit evidence and this runbook. Keep ingress closed until checks pass:
|
||||
|
||||
@@ -128,11 +135,21 @@ The explicit new verification run reads the manifest, `driver.json`, migration/l
|
||||
|
||||
Prior post-record live-size samples remain manager ack **0.6078 s**, registration catch-up **0.0128 s**, mail.send-to-default publication **0.5376 s** (integrated 0.5235 s), below existing orchestrator **3 s** / cred **10 s** caller budgets. Disclose integrated catch-up **12.0419 s** and the new initial-scan delay above. They are synthetic/warm-cache observations, not production latency guarantees. Prior real-core watch/upgrade/all-watch readiness, ~0.10% whole-host idle duty, detached error data and real cron warning/stop evidence remain linked in the result docs.
|
||||
|
||||
Before any Git network work, record locked-startup proof, ask **Surma to unlock the target vault through the existing UI**, then check `cred.status` reports unlocked for that same target service. Wait here if Surma is unavailable; never transfer a key/password. **Only after that**, mediator/orchestrator activates the reviewed git artifact:
|
||||
|
||||
```json
|
||||
{"method":"core.activate","params":{"plugin":"git","wasm":"/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm"}}
|
||||
```
|
||||
|
||||
This is a proposed live operator step covered by the go, not a worker action. The permanent world artifact above must be staged/hash-checked from the selected bundle before go; the driver preflights its external `deferred_artifacts.git` copy too. Verify active git hash equals manifest `files["git.wasm"].sha256` before releasing Git callers. See `/plugins/git/source/{VAULT-HTTPS.md,VAULT-SSH.md}`; both require cred first, live and unlocked. No push is part of verification; **every** later push still needs a separate fresh Surma go. Final proof includes `git_activated_after_unlock`.
|
||||
|
||||
Interim compatibility: observed live Git **f089b56d** still speaks new cred's explicit-grant fetch/item_get/public_key/sign contracts. Actual old-WASM + new-cred loopback SSH clone signed/authenticated successfully with a synthetic explicit grant after unlock; locked use returned vault_locked. Granted HTTPS reached new cred, reserved exactly one use, then the deliberately closed loopback port returned network_error with a capture (no successful HTTPS transfer claimed). Old grant-free shared SSH fails asking for a local key; old grant-free HTTPS retains its old config/anonymous path, not new anyone_may_use selection. Therefore do not schedule Git network work during the locked/old-git interval, do not introduce local key/config-secret workarounds, and surface any already-started caller's lock/missing-key error rather than blindly retrying it. Local git status/commit/authoritative-main merges remain available. New Git shared SSH clone was tested after real synthetic unlock/activation.
|
||||
|
||||
Only now use **`orchestrator.release` for SAU-375** to release its approved commit. Inspect that method's current typed arguments; do not invent them. Verify published asset hashes and ingress. No worker publishes anything. The rework tested approved asset hashes in temporary output, **not an actual live release or physical browser**.
|
||||
|
||||
After recording the required **locked** startup proof, the new mediator may need Surma to unlock the new vault through the existing UI before the approved signing grant can support the final SSH command. Plan for that explicit post-start unlock; never transfer the old vault key or ask for its password. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain `awaiting_verification`, keep ingress/release gated, and do not reconstruct or rerun the migration.
|
||||
The Surma unlock above also enables the existing approved signing grant for the final external SSH verification/deletion command; it is not a second vault handover. Never ask for Surma's password or transfer the old vault key. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain `awaiting_verification`, keep git activation/ingress/release gated, and do not reconstruct or rerun the migration.
|
||||
|
||||
Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command:
|
||||
Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`, `git_activated_after_unlock`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py verify operator.json verification.json 2>> tool-source-and-delete.log
|
||||
@@ -144,11 +161,41 @@ It marks the saved plan/driver verified, logs source hashes/source-as-log, and *
|
||||
|
||||
| Boundary | Exact recovery rule |
|
||||
|---|---|
|
||||
| Waiting for arm | World untouched. Inspect the saved controller identity before cancelling; do not launch another start against existing checkpoints. |
|
||||
| Waiting for arm, including consumed-marker/save race | World untouched. Repeated arm is finite even with no reader. After the saved controller exits, `resume` accepts waiting_for_arm and the durable marker. Or run abort (untouched) after obtaining the external controller lock. |
|
||||
| Armed/captured/owner_stopped/frozen; no target attempt | External operator can use `... driver.py resume operator.json` **only after the saved controller has exited**. It detaches a replacement; same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed. |
|
||||
| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively restore the full frozen backup coherently and launch the explicit old-owner wrapper. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
|
||||
| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively Surma runs the real driver's **abort** mode below: it restores every approved changed path from the immutable complete backup, verifies the full frozen inventory, and launches the explicit old-owner in the selected pane. It never restores an incomplete backup. If planning never completed, apply cannot have started, so it verifies any saved inventory and relaunches the untouched old world. Restore is before/after-hash guarded, resumable, and writes main DBs before their old sidecars without opening them. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
|
||||
| `started` or later, even if boot seems failed | No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them and plan a narrow forward repair or coordinated revert. |
|
||||
| Any plugin core.revert | Offline-swapped prev now means exactly pre-cut current (hash-tested), not pre-pre-cut. **Do not use blind individual core.revert across incompatible contracts.** Coordinate providers/callers and preserve new work; the session prerequisite below still applies. Later git activation uses normal core prev semantics. |
|
||||
| Session revert after target activity | Close gateway/ingress. While new manager is available, **unsubscribe web-chat**, verify removal, and prevent gateway re-registration while the old session binary runs. Never call web.start against that old manager or restore a registry backup that reintroduces it. Actual old-manager/gateway tests proved unsubscribe safe and ID-only re-key unsafe: the old binary ignores best_effort and replays history once. |
|
||||
| Host reboot / changed PID identity / unexpected survivor | Refuse saved native identities; re-inventory externally. Never signal a reused PID, blindly kill a group, or infer quiescence from a stop request. |
|
||||
|
||||
### Exact external failure commands (Surma, not an in-world mediator)
|
||||
|
||||
In the independent Citadel terminal, `cd /home/surma/sau-384-cutover`; read `driver.json`, `failure.json`, `driver.log` and the pane once. Commands use the pinned interpreter:
|
||||
|
||||
```sh
|
||||
PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
|
||||
# Same manifest/boot/owners, known exited controller, pre-start only:
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py resume operator.json
|
||||
# Or abandon pre-start attempt, restore coherent old bytes, submit old-owner once:
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py abort operator.json
|
||||
```
|
||||
|
||||
Choose **one**, not both. Both use a nonblocking exclusive controller lock; do not delete the lock file or bypass it. Abort accepts waiting/armed/frozen/installed/restoring/restored only. It checkpoints `old_start_attempted` **before** pane launch, then `old_launched`. If interrupted during restoring/restored, repeat abort on the same manifest/checkpoints; if `old_start_attempted`/`old_launched`, inspect the pane instead of repeating it. Abort is forbidden after any target `started` attempt. Never call Kit.rollback by hand or relaunch a bare loop. If the pinned shell/pane changed, stop for explicit identity resolution; do not guess a replacement.
|
||||
|
||||
After abort, Surma verifies exactly one old package owner in the pane and old world readiness before retry. Preserve the frozen evidence, never delete it to make a check pass. For a clean retry **only after old_launched and verification**, archive checkpoints (not owners/selectors/GC roots/tool):
|
||||
|
||||
```sh
|
||||
# Pick a fresh, unused archive path and record it; this example is this run.
|
||||
ARCHIVE=attempt-before-retry
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B -c 'import json; assert json.load(open("driver.json"))["phase"] == "old_launched"'
|
||||
mkdir -- "$ARCHIVE"
|
||||
for p in driver.json freeze.json arm.json controller-process.json migration driver.log failure.json failure-notification.json preflight-live.json; do
|
||||
if [ -e "$p" ]; then mv -- "$p" "$ARCHIVE/"; fi
|
||||
done
|
||||
cp -- operator.json "$ARCHIVE/operator.json"
|
||||
```
|
||||
|
||||
Re-identify the now-running old loop/host and pane shell; refresh old-work/caller guards (including any wake-up) and approval for the new resolved `operator.json`, retaining the stable continuation UUID if no target start was attempted. Then check/start/arm as §1. Do not merely edit a live checkpoint or delete the backup. For aborted_untouched, no old relaunch is necessary; archive only after verifying the original owners are still running. No such reset/archive is allowed for started/awaiting_verification/verified or unknown owners: retain new work and plan a forward repair.
|
||||
|
||||
Independent plugin recovery/accounting/provider-cancellation follow-ups remain independent. Accepted first-update cuts and uncertain external effects are not silently solved by this kit, grace=5, or a successful rehearsal.
|
||||
|
||||
Reference in New Issue
Block a user