Rehearse shared-disk failures and forward-only cutover completion
ci / check (push) Failing after 3m24s
ci / check (push) Failing after 3m24s
Card: SAU-384
This commit is contained in:
@@ -6,6 +6,8 @@ Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover
|
||||
|
||||
## Closed integration gaps
|
||||
|
||||
- **Round 4, run 1392:** allocated-block/directory/sparse-copy space budgeting; disk-independent foreground progress/failure/pidfd-death monitor; incomplete pre-plan backup cleanup before abort writes plus low-space refusal; forward-only post-start continue with durable locked proof and stable continuation ID; guarded atomic-temporary cleanup; converter dry-run before freeze. Shared-filesystem ENOSPC (world/root/log at zero free bytes), each post-start RPC failure including lost committed send reply, real 120 s registration timeout, partial atomic crash/abort, wrapper-exec handshake/resume, sidecars and full workspace passed. Tools deleted after verified success. Source/logs, fixture failures and limits: [rework-1392.md](rework-1392.md). Live plan is still approval-gated.
|
||||
|
||||
- **Round 3, run 1389:** deferred approved git 9cc9fa6f is bundled/preflighted, activated only after target cred is live and Surma unlocks. Real old/new Git + new cred loopback tests covered granted SSH, locked denial and new shared SSH. Added real-driver pre-start abort/restore/old-owner relaunch, live consistent-copy/executable-only caller guards, special-file/free-space gates, finite registration deadline/failure notification, exact pre-cut prev images and durable arm-marker crash recovery. Real partial-swap/ENOSPC/audit failures, arm interruption, final Herdr cutover/preservation/sidecars and full workspace (208 host tests) passed. Tool/controllers deleted after verification. Exact logs, selection and limits: [rework-1389.md](rework-1389.md); proposed live recovery/roots-side-effect plan: [runbook](runbook.md).
|
||||
|
||||
- **Review rework, run 1381:** one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See [rework-1381.md](rework-1381.md) for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). [Runbook](runbook.md) now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# SAU-384 round 4 — run 1392
|
||||
|
||||
Ready for review; **not live execution approval**. Addresses all five findings in extra-round-4.md. No shipped Rust/plugin behavior changed. Current-main source and state contracts were rechecked; all work was in the card tree, review artifacts and disposable box tc-zg0j56.
|
||||
|
||||
## Selection and operator plan
|
||||
|
||||
The selected bundle is unchanged: `cutover-targets-1389.tar.gz`, **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**, 23 payloads. All payload sizes/hashes and the native closure hashes were freshly checked. Host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0` from e09c4f9f. Cred remains 0db40101 / build 1636 (274 tests); mail/cron/rooms remain 1600/1602/1603. Deferred git remains approved 9cc9fa6f, activated only after target cred is live and Surma unlocks. No new WASM/package build. SDK-copy list: **src/lib.rs only**. Approved web 3d760267 remains held until the operator releases SAU-375 after session/web and verification prerequisites.
|
||||
|
||||
Exact commands and approval boundaries are in [runbook.md](runbook.md). Current non-executable operator template: `evidence/round4-1392/live-manifest.template.json`. It still requires fresh caller/owner identities, guards and explicit Surma go. The only executable source is reconstructable **as log**, not retained as a tool. Surma is the external human recovery operator. The old mediator is deliberately cut; the driver publishes a new explicit default verification input, not inferred recovery.
|
||||
|
||||
## Five findings fixed
|
||||
|
||||
1. **Shared-filesystem ENOSPC.** Space preflight measures POSIX allocated blocks for files/directories/root/non-followed links, and charges dense filesystem-fragment-rounded size where copy2 would expand a sparse file. It includes the concrete staging/prev/SDK/DB-copy roles; receipts save all measurements. Removed the false never-underestimates claim: metadata/normalization/growth/competing writers remain uncertainty. A separate sparse/tiny/directory fixture copied **4,599,808 allocated bytes** from **12,288 source-allocated bytes**, below its measured **4,603,904-byte** budget. No live margin/default was invented.
|
||||
|
||||
Mandatory foreground `driver.py monitor` receives memory-only Unix datagrams for inventory/backup progress and failure, and kernel pidfd controller-exit completion. Every start/resume waits for its own monitor; arm requires it. Failure JSON/log writes are best effort and cannot prevent CUTOVER FAILED or the independent Herdr notification attempt. A terminal still reports controller death on SIGKILL; no heartbeat/polling loop. Notification submission was tested, but returned shown:false/disabled; physical display is not claimed.
|
||||
|
||||
Actual ENOSPC used **one shared tmpfs for world, external root and driver.log**, device 83, **0 free bytes**, capacity 254,504,960 bytes derived from measured fixture/budget allocations. It failed during a real backup write after good preflight. Monitor showed failure and death despite unwritable disk evidence. Pre-plan abort removes only the reproducible incomplete backup subtree before any checkpoint write, retaining planning/inventory evidence. If the competing writer still fills the disk, it refuses before saving/launching. After that known test file was freed, the same abort restored coherent old ownership and readiness. Complete backups are never deleted. Every abort (with or without a completed plan) now checks space before its first checkpoint save; a second shared-filesystem zero-free-byte fixture proved the complete backup/checkpoint remained exact, then old relaunch worked after freeing only the known competing test file. Space is checked again before old relaunch. This is not proof against every filesystem's ENOSPC/power-loss behavior.
|
||||
2. **Post-start forward completion.** New `continue operator.json` locks, checks the bound manifest, exact pane/shell, exactly one target host and target plugin hashes, and performs only missing/read-retry-safe post-start RPCs. It never launches/applies/restores. Actual locked cred.status is saved durably before unlock; subscribe uses the stable cutover registration; send reuses the exact continuation UUID/text. It reaches awaiting_verification, then normal verify/delete is reachable. Real failures of **cred.status, subscribe, and send**, including send committed before its acknowledgement was lost, all recovered with one run-start/consumption. A real **120 s** absolute registration timeout recovered the same way without another owner. The value remains cron/src/types.rs's existing registration/publication bound. Runbook gives exact continue command, RPC payload derivation, locked-proof/unlock ordering and local Surma fallback if live SSH signing is unavailable.
|
||||
3. **Atomic temporary cleanup.** A child was SIGKILLed inside atomic after writing a real partial plugin replacement temporary, before rename. Abort removes only a saved operation's newly created .cutover-tmp, absent in frozen inventory and prefix-matching an approved before/after image; unknown bytes/links refuse. Full frozen inventory equality passed inside rollback before old launch. Resume still overwrites/completes its own temporary. No arbitrary world cleanup.
|
||||
4. **Converter pre-freeze dry run.** check/start, arm and immediate pre-freeze checks use the same pure cron converter as plan, on SQLite backup-API memory copies. interrupt:true, invalid text and non-object params refuse without freeze/mutation. Negative control: round-3 checker accepted the same invalid object payload, new checker refused. No compatibility code in a plugin.
|
||||
5. **Roots/wake-up documentation.** Every controller resume repeats all three roots queries and their possible stale-registration pruning; this remains part of the proposed Surma-approved live plan. This round queried roots only against a private rehearsal Nix DB. No cron wake-up is needed: the stable explicit driver continuation owns handoff. A due-during-downtime job is skipped; a just-before-freeze input can enter an old pending queue that is intentionally dropped. Neither is replayed. The runbook disallows relying on guessed backup duration as readiness.
|
||||
|
||||
Testing also exposed a real **Nix Python wrapper exec identity race** in parent-side process capture. The launcher now receives the child's actual post-wrapper identity over a pipe-completion handshake. Controller kill/resume proved exact identity, a fresh monitor requirement and successful same-pane continuation. No liveness/endpoint polling was added.
|
||||
|
||||
## Full logs and evidence
|
||||
|
||||
All log names below have prefix `/logs/box/2026-10-31/tc-zg0j56-exec-`. Evidence is `evidence/round4-1392/`.
|
||||
|
||||
| Gate | Full stderr / outcome |
|
||||
|---|---|
|
||||
| Final shared world/root/log ENOSPC, pre-save incomplete-backup deletion, low-space refusal, old readiness; partial atomic SIGKILL/abort; all three post-start RPC failures; real registration timeout | **166388571.stderr**, clean combined pass on final tool source. Evidence under `final/`, including shared-enospc.json, atomic-abort.json, *-reviewed.json and registration-timeout.json. |
|
||||
| Complete-plan ENOSPC on one shared filesystem: pre-write refusal, checkpoint/full-backup equality, old readiness after known competing-data cleanup | **4094731865.stderr**, pass; `final/complete-plan-enospc.json`. |
|
||||
| Controller wrapper handshake, killed-controller pidfd, resume waits for fresh monitor, target completion | **537908850.stderr**, pass; `resume-handshake.json`, both monitor logs. |
|
||||
| Converter pre-freeze refusals | **1954198187.stderr**, pass; `dryrun.json`. |
|
||||
| Negative control + sparse allocation + all payload/native closure hashes + **133** final source/build inputs | **1314968883.stderr**, pass; `converter-negative-control.json`, `allocated-space.json`, `source-and-bundle.json`. |
|
||||
| Committed WAL/hot rollback journal; exact pending/cancel/mail losses; preserved mail/jobs/IDs/calendar/outcomes/non-target JSON; apply interruption/rerun and physical rollback | **2622539348.stderr**, pass; `sqlite-sidecars.json`. |
|
||||
| cargo test --workspace, unchanged final production source | **3315145279.stderr**, pass; **208 host integration tests**, result at line **433**, plus all workspace suites/doc tests. |
|
||||
| Fresh target one-manager/locked-cred/original-byte preservation/pre-cut-prev verification; actual verify CLI and complete tool/test deletion | **3030415406.stderr**, pass; `final/` final-ownership-preservation.json, verification.json, final-verified-driver.json, tool-deletion.json, test-deletion.json. Earlier verify/deletion pass: 1403220230.stderr. |
|
||||
| In-memory source-log SHA/AST reconstruction; executables/bytecode still absent | **2019074395.stderr**, pass; `final/source-log-roundtrip.json`. Earlier docs/JSON/absence checks: **2041020851.stderr**; final docs/JSON/absence/23-payload/133-source-input validation: **1011452644.stderr**, `final/final-validation.json`. |
|
||||
|
||||
### Executed source deleted after verified success
|
||||
|
||||
Reconstruct only these three files from **3030415406.stderr** for a later approved live operation; do not reconstruct tests:
|
||||
|
||||
- cutover_once.py: source lines **2–189**, SHA-256 **84b13644bd80c6b331dcd8df9bf79339f62fedb99f7dd376535b2bca7b0afd96**;
|
||||
- freeze.py: source lines **192–263**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
|
||||
- driver.py: source lines **266–669**, SHA-256 **5c3cba1a69b8b5e8f82ea83caa0697af494bf93dd0297f0c1b5231192cbb5fad**.
|
||||
|
||||
After generalizing the no-write space gate to complete-plan and untouched abort too, the source was reconstructed for the final affected rerun, then the real verify command again marked that rehearsal verified, logged/deleted the three files and caches; all executed/reconstructed tests/patches/wrapper were logged/deleted too. Permanent owner/selector installation artifacts are distinct, not migration code. Live runbook still mandates deletion after verified success.
|
||||
|
||||
## Failed approaches and limits
|
||||
|
||||
- First interpreter alias linked only bin/, losing Nix Python's site-package context. Linked the complete existing interpreter environment; imports and gates then passed.
|
||||
- Cold workspace cache made six offline guest builds miss async-task; populated the disposable cache and reran the **whole workspace**, 208/208 host tests, unchanged source. Initial full log **2236066973.stderr**, missing dependency at lines 755–785; final pass above. No retry-without-diagnosis claim.
|
||||
- First shared filesystem test refused **before freeze** because actual source growth exceeded its provisioned measurement. Recorded that valid negative control, then provisioned the test from measured fixture/budget roles before inducing deliberate post-preflight pressure. A later test filled before the backup had copied any data: abort correctly needed a no-write space check before its first save; fixed and repeated with a before-backup event gate. Final shared test reached zero free bytes during copy and the final combined gate passed cleanly. Earlier fixture checkpoint-name collisions, an existing evidence-export directory and a busy mount affected cleanup/export only; used fresh identities/paths and explicit disposable-pane cleanup, never edited migration checkpoints. Mounts were left for disposable-box teardown, not live cleanup.
|
||||
- First post-start fixture omitted session prompt files, so it correctly recorded run-end error rather than mailbox consumption. Added synthetic prompts and repeated all three failure modes from fresh worlds; each then produced exactly one consumed continuation.
|
||||
|
||||
This round's failure fixtures are **small synthetic real-core/Herdr worlds**, not another live-size benchmark. Prior accepted live-size race/replay/core-watch/upgrade/deadline/idle/grace/crash-supervision, scale and Git compatibility/unlocked activation evidence remain cumulative. Startup 0.6078 s, catch-up 0.0128 s, mail publication 0.5376 s vs callers' 3 s/10 s limits, catch-up outlier 12.0419 s and initial-scan delay remain disclosed in earlier reports; not remeasured or upgraded to guarantees here.
|
||||
|
||||
Unverified/out of scope: actual approved live cut-over/frozen caller audit, live detached SSH/Herdr w7:p6 execution, live SSH signing through target cred, Surma's real unlock/new-git external networking, physical notifications/browser, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. Surma's external terminal has exact continue/verify commands, so final completion is not dependent on the untested live signer. No push, live activation/restart/config/owner change, release, deployment or /www publication. Surma go still precedes mediator core/session/cred operation, the proposed detached-SSH exception/root side effects, and **every push**.
|
||||
+57
-24
@@ -1,6 +1,6 @@
|
||||
# One-off handover runbook — rehearsed; live approval required
|
||||
|
||||
Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md), [round-3 recovery/git gates](rework-1389.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
|
||||
Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md), [round-3 recovery/git gates](rework-1389.md), [round-4 failure paths](rework-1392.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
|
||||
|
||||
## 0. Prepare everything while the old world is still available
|
||||
|
||||
@@ -17,11 +17,11 @@ Do not begin downtime with scripts, caller decisions or launch instructions stil
|
||||
|
||||
### External driver and resolved operator manifest
|
||||
|
||||
Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-asa3yr-exec-2013747101.stderr`; check their hashes against `evidence/round3-1389/tool-deletion.json`. These are the executed/reviewed round-3 versions, not the earlier scripts from runs 1373/1381. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
|
||||
Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-zg0j56-exec-3030415406.stderr`; check their hashes against `evidence/round4-1392/final/tool-deletion.json`. These are the executed round-4 versions (source lines 2–189, 192–263, 266–669 respectively), not earlier scripts. Hashes/ranges are in evidence/round4-1392/final/source-log-roundtrip.json. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
|
||||
|
||||
The tested interpreter is `/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3`, with **inotify-simple 2.0.1**. Resolve its closure availability before go. After go and before starting the driver, pin its package and the inspected Herdr package with host-visible GC roots for the operation (`nix-store --realise <exact-package> --add-root <external-root>/gc-controller --indirect`, and equivalently `gc-herdr`). These root writes/imports are included in the approved live plan, not preparation actions authorized to a worker. The driver itself registers and verifies **old host, target host and loop** roots at `<root>/gc-{old,target,loop}` before freeze. Use the **host namespace**, not a box talking to shared Nix state: `/rework` roots in a box are not host-visible. Do not run GC as a test. Keep old/target/loop roots and both selectors for recovery after tool deletion.
|
||||
|
||||
Start from `evidence/round3-1389/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
|
||||
Start from `evidence/round4-1392/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
|
||||
|
||||
- native paths, Herdr binary/context, pane and current shell/loop/host PIDs;
|
||||
- external root, world, bundle, embedded manifest SHA, offline artifact paths **and deferred git**, SDK path, exact target boot list; root and world are on the same filesystem in this plan; resolve a different storage topology before go;
|
||||
@@ -29,7 +29,7 @@ Start from `evidence/round3-1389/live-manifest.template.json`, a **non-executabl
|
||||
- a new continuation UUID, generated **once**, and the explicit new mediator verification instruction. Keep that UUID on all retries;
|
||||
- typed executable-caller review and its frozen guards below.
|
||||
|
||||
Do not copy sandbox PIDs, `gc_store`, or `test_gate_after_freeze` into a live manifest. The template deliberately omits those test-only fields. Record/hash the fully resolved `operator.json`; driver checkpoints bind to its exact bytes. Resolve a schema/selection discrepancy while the old world is available, not during downtime.
|
||||
Do not copy sandbox PIDs, `gc_store`, or any `test_*` gate into a live manifest. The template deliberately omits those test-only fields. Record/hash the fully resolved `operator.json`; driver checkpoints bind to its exact bytes. Resolve a schema/selection discrepancy while the old world is available, not during downtime.
|
||||
|
||||
### Executable caller review and frozen guards
|
||||
|
||||
@@ -39,9 +39,15 @@ Use predecessor formats: session source `f9122585` Paths/queue; cron `ba75c102`
|
||||
- board `state/board/board.sqlite`: ordered `id,data` for hooks/invariant_target; `id,target,session,data` for subscriptions; and `select id,data from subscriptions union select subscription,data from subscription_batches order by 1,2` for executable subscription snapshots. Ignore last_sent/ended, transient batch IDs/lines/due; a pending delivery of the same active configuration does not change this union. Historical hook-attempt snapshots remain diagnostics and **must not replay**. Automatic final subscription removal changes the executable set; resolve/refresh before arm, or take the pre-start abort path if it happens after arm;
|
||||
- boot and other discovered executable configuration/manager registrations. Review typed room/workflow/orchestrator stores too; preserve their histories/settings and non-executable records. Do not mistake prompt text for an RPC template.
|
||||
|
||||
A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. These projections, including all column names, live in the round-3 template and were tested against mutable cron/board bookkeeping. Any genuinely new executable store/field still needs review; do not silently omit it. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
||||
A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. These projections, including all column names, live in the current template and were tested against mutable cron/board bookkeeping. Any genuinely new executable store/field still needs review; do not silently omit it. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
|
||||
|
||||
**Before start, again before arm, and once immediately before freeze**, `check`/driver checks artifact and file guards, SQL projections on consistent live SQLite backup-API copies, inventory special-file types (without hashing mutable bytes), and available space. The space budget is the measured current regular-file logical bytes plus staged plugin/prev/SDK bytes and the concrete recovery/stage/audit DB copies; the receipt saves required/free bytes. Root/world free space is checked before downtime; further log growth, changed mounts or disk exhaustion can still occur. After freeze the driver backs up all bytes, checks these projections on **disposable complete DB copies**, and verifies file guards against the frozen backup **before apply**. It never recovers/opens immutable backup evidence in SQLite. A changed frozen projection stops before apply with `failure.json`, log and a best-effort Herdr notification. Surma uses the exact pre-start abort command below; there is no in-world human gate after stop-all. Include any newly discovered executable store in the approved manifest before arm. Add the mediator's planned cron wake-up **before resolving/hash-approving operator.json**, never between resolution and freeze. From resolution until freeze: no job add/remove/template/schedule edits, hooks/subscription/invariant/manager-registration changes, boot/SDK/plugin activation or other executable-caller changes. Routine next/last/result and subscription delivery bookkeeping is permitted. If new intent is needed, cancel unarmed or abort pre-start, re-resolve, and obtain approval for the changed manifest; do not edit a hash-bound in-progress manifest.
|
||||
**Before start, again before arm, and once immediately before freeze**, `check`/driver checks artifact and file guards, SQL projections on consistent live SQLite backup-API copies, special-file types (without hashing mutable bytes), space, and a **dry run of the same cron converter used by plan**. Object/text/interrupt refusals therefore occur before downtime, not just in plan. The dry run never rewrites the live DB. After freeze, guards run on disposable complete DB copies before apply; immutable backup evidence is never opened/recovered in SQLite.
|
||||
|
||||
**Space is an allocated-block measurement, not a guarantee.** Root/world must have the same filesystem device in this reviewed layout. Walk regular files, directories (including the root) and non-followed symlinks; charge each at the larger of POSIX `st_blocks × 512` and its logical size rounded up to that filesystem's `f_frsize`. This includes sparse-file expansion by copy2, not just logical file bytes. Add the concrete staged plugin/prev/SDK allocations and the three existing DB-copy roles (recovery, staging, audit). Save source allocation, backup allocation budget, directory/file counts, fragment size and required/free bytes in `preflight-live.json`; refuse before freeze unless free covers that budget. The fresh sparse/tiny-file/directory fixture measured 12,288 source-allocated bytes and 4,599,808 copied bytes against a 4,603,904-byte budget (see evidence/round4-1392/allocated-space.json for actual measured values). **No claim that this can never underestimate:** metadata, DB normalization, growth, changed mounts and competing writers can still exhaust space. Resolve storage uncertainty in the approved maintenance plan, not with an invented margin.
|
||||
|
||||
Add any genuinely new executable store to the manifest before arm. From resolution until freeze: no caller/boot/SDK/plugin changes; routine cron next/last/result and subscription delivery bookkeeping is permitted. If new intent is needed, cancel unarmed or abort pre-start, re-resolve, and approve the changed manifest; never edit a hash-bound in-progress manifest.
|
||||
|
||||
**No cron wake-up is required:** the external driver publishes the explicit new mediator input. Do not rely on a timed wake-up for controller recovery. If a fallback wake-up is explicitly wanted, add/review it **before** resolving operator.json, not between resolution and freeze. It is useful only after the target is running and can accept new input. One due during downtime is skipped by cron; one firing immediately before freeze can enter an old pending queue that this approved cut deliberately drops. Neither is migrated/replayed. A guessed wall-clock time after the estimated backup duration is not readiness proof; use the driver's event/completion evidence and stable continuation instead.
|
||||
|
||||
### Approval and external launch surface
|
||||
|
||||
@@ -51,7 +57,11 @@ The proposed concrete execution surface is **two finite foreground `ssh.exec` ca
|
||||
|
||||
## 1. Surma monitors; mediator starts/arms the external controller
|
||||
|
||||
Close ingress and record accepted first-update cuts before arm. **Surma is the external human recovery operator while the world is down.** Before arm he opens an independent Citadel terminal, changes to `/home/surma/sau-384-cutover`, and runs `tail --follow=descriptor driver.log` (GNU tail on this local filesystem uses inotify; no log rotation during the attempt) after start creates it. This terminal must not be an in-world session/tool process. The driver writes `failure.json` and emits `CUTOVER FAILED` on failure, and submits one best-effort Herdr notification `Cut-over needs Surma`; notification submission is saved in `failure-notification.json`, but no connected-client display is guaranteed. Terminal monitoring remains mandatory (SIGKILL/power loss cannot send a notification). Inspect checkpoint/controller identity once if the controller is unexpectedly gone, never repeatedly poll.
|
||||
Close ingress and record accepted first-update cuts before arm. **Surma is the external human recovery operator while the world is down.** He uses an independent Citadel terminal, never an in-world session/tool process. After mediator `start` acknowledges, Surma runs the foreground monitor below and leaves it visible. It binds a local Unix datagram socket and records the monitor/controller identities before arm. The controller waits for **its own** monitor on every start/resume; arm refuses without it. If an old monitor was killed, a new monitor removes its stale socket only after the saved exact process identity is dead; never unlink a live monitor's socket by hand.
|
||||
|
||||
The monitor receives inventory begin/completion and completed backup-entry progress **without writing disk**, plus `CUTOVER FAILED` and the notification outcome. It also waits on the controller's **pidfd**: normal exit, SIGKILL or crash prints `CONTROLLER EXITED`, even when there is no log/failure.json. Progress datagrams are best effort; the independent death event is not. No heartbeat/liveness/endpoint polling or backup-duration timeout. Read `driver.json` and inspect the pane once after exit; distinguish awaiting_verification from failure instead of blindly restarting.
|
||||
|
||||
Disk-backed `failure.json`, `failure-notification.json` and driver.log are best effort. Each may be unwritable at ENOSPC; none is a prerequisite for the failure signal or the Herdr notification attempt. One best-effort Herdr notification `Cut-over needs Surma` is submitted without first saving failure.json. Rehearsal submission returned shown:false/disabled; neither delivery nor a physical display is guaranteed. **Keep the external terminal monitor attached:** a silent tail of driver.log is not sufficient during backup. Whole-machine power loss can stop both controller and monitor and is not claimed covered.
|
||||
|
||||
From the external root, on the approved host execution surface, mediator first runs the finite preflight command, then starts:
|
||||
|
||||
@@ -60,13 +70,20 @@ PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py start operator.json
|
||||
```
|
||||
|
||||
`start` launches a new OS session with stdin closed and output in `<root>/driver.log`, records its PID/start identity, returns an acknowledgement, and leaves the controller waiting on an inotify watch for **durable arm.json**. There is no arm FIFO or reader-dependent write. Check that receipt and checkpoint once, then:
|
||||
`start` launches a new OS session with stdin closed and output in `<root>/driver.log`. A pipe completion handshake records the child's **actual post-Nix-wrapper-exec identity**, rather than racing an interpreter exec. It returns an acknowledgement and leaves the child waiting on inotify for its own monitor, then **durable arm.json**. There is no arm FIFO or reader-dependent write. Check that receipt/checkpoint once. Surma now runs in his independent terminal:
|
||||
|
||||
```sh
|
||||
cd /home/surma/sau-384-cutover
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py monitor operator.json
|
||||
```
|
||||
|
||||
Wait for `MONITOR attached; arm may now proceed` in that terminal. Then mediator runs the finite arm command:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py arm operator.json
|
||||
```
|
||||
|
||||
`arm` repeats live preflight and atomically writes a manifest-bound marker; re-arming is idempotent while waiting, including when no controller is alive. A crash after observing the marker but before saving `armed` resumes from `waiting_for_arm` and reads the same marker.
|
||||
`arm` requires the current monitor, repeats live preflight and atomically writes a manifest-bound marker. Re-arming is idempotent while waiting with that monitor; without it arm refuses finitely, never blocks on a missing FIFO reader. A crash after observing the marker but before saving `armed` resumes from `waiting_for_arm` and reads the same marker.
|
||||
|
||||
From here **the external driver, not the old mediator**, owns steps 2–4. The old default run and its tool calls are deliberately cut. They are not recovered from legacy history. The driver uses an external advisory lock to exclude another controller. Rehearsal killed both the invoking caller shim and, separately, the detached controller after freeze; checkpoint resume completed the chain.
|
||||
|
||||
@@ -76,9 +93,9 @@ The driver verifies reviewed bundle/plugin/SDK bytes and native closure hashes,
|
||||
|
||||
1. Pins boot ID, start ticks, executable and group for both native processes; confirms controller is outside their group. Signals the **outer loop first** and awaits its pidfd exit. Saves `owner_stopped`.
|
||||
2. Signals the exact host **SIGTERM then SIGCONT**, so a job stopped by background-PTY SIGTTIN can handle termination, and awaits host exit. Checks the saved group once for unexpected survivors; never blindly kills them. Saves `freeze.json` phase **`frozen`**, world path, boot ID and empty group inventory. No process/endpoint polling, sleeps or stop-request counting. Host shutdown stops all plugin/agent trees and suppresses restart admission. `core.restart` alone is not stop-all; after the loop is gone, exit 75 cannot respawn it.
|
||||
3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks **without following links**. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Unknown images refuse.
|
||||
3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks **without following links**. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Progress is sent to the external terminal monitor. A pre-plan abort may discard only that reproducible incomplete backup, as described below. Unknown images refuse.
|
||||
4. Stages the approved transformations, then checks the frozen caller guards. **Both plan and apply independently require the actual frozen checkpoint**, same world/current boot, dead pinned owners, empty old group and no subsequently running host for this world. A caller-supplied `{remaining:0}` literal is not accepted. Changed checkpoint bytes also refuse.
|
||||
5. Stages **plugin.prev.wasm = that plugin's frozen pre-cut plugin.wasm** for every offline-swapped plugin, then target plugin.wasm. Both prev/current writes have before/after hashes and participate in restore. It never retains the unrelated older prev as the cut-over fallback. Git is deliberately untouched here; its later standard activation will save its own preceding binary. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. `migration/plan.json` reaches `installed` only after all operations complete.
|
||||
5. Stages **plugin.prev.wasm = that plugin's frozen pre-cut plugin.wasm** for every offline-swapped plugin, then target plugin.wasm. Both prev/current writes have before/after hashes and participate in restore. It never retains the unrelated older prev as the cut-over fallback. Git is deliberately untouched here; its later standard activation will save its own preceding binary. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. If atomic replacement dies before rename, abort removes only that saved operation’s new `.cutover-tmp`, absent from the frozen inventory and byte-prefix-matching its approved before/after image; unknown temporaries refuse. Resume overwrites/completes its own temporary. `migration/plan.json` reaches `installed` only after all operations complete.
|
||||
|
||||
Exact logical losses are **only** old session `pending.jsonl`, `cancel`, and unclaimed `mail.deliveries` rows with `state='pending'` and recipient not `surma`. Original messages, Surma copies, done rows and uncertain `unknown` deliveries stay; do not replay them. Loss records and original bytes/rows remain in backup/evidence. Histories/settings, scheduled SQL IDs/calendar/outcome fields, grants/requests and unrelated files remain intact.
|
||||
|
||||
@@ -106,17 +123,31 @@ Manager precedes web, so `web-chat` becomes best-effort. No `mail.start` deliver
|
||||
|
||||
The permanent `<root>/target-owner` exports `SAUSAGE_FLAKE=path:<root>/target-selector` itself, then execs the inspected loop with the exact world path. The selector's sole app is the pinned target package. `<root>/old-owner` similarly selects the **explicit old package**, never the remote default. These are permanent installation/launcher artifacts, **not migration code**. Their bytes/hashes are retained as manifest evidence; do not delete them with `tool/`.
|
||||
|
||||
**Every later pane/cold restart uses `/home/surma/sau-384-cutover/target-owner`, with no caller environment required. Never invoke bare sausage-loop or the mutable gitea default for this world.** Keep this command in the operator's pane-restart procedure. Rehearsal cold-launched it in the same Herdr shell without ambient SAUSAGE_FLAKE and observed the target executable. The explicit old wrapper was executed only against a separate old-backup clone.
|
||||
**Every later pane/cold restart uses `/home/surma/sau-384-cutover/target-owner`, with no caller environment required. Never invoke bare sausage-loop or the mutable gitea default for this world.** Keep this command in the operator's pane-restart procedure. Rehearsal cold-launched it in the same Herdr shell without ambient SAUSAGE_FLAKE and observed the target executable. Earlier cold rollback exercised the explicit old wrapper against a separate old-backup clone; pre-start abort gates also exercised it against the exactly restored original frozen fixture, never over new target work.
|
||||
|
||||
`core.restart` now restarts the pinned reviewed host; it does **not** self-update from gitea. Future host updates require a separately approved package/closure/root/selector change. A remote push alone must never silently choose an old host against target state. Keep both GC roots through the rollback window; retire the old root only after an explicit decision, not tool deletion.
|
||||
|
||||
**Expected approved live Nix side effect:** the driver makes three `nix-store --query --roots` calls, one each for old package, target package and loop, immediately after rooting those paths and before freeze. Those queries can prune stale auto-root registrations and stale temporary-root files in Nix state, even though they do not delete store packages. Surma's go must include that side effect (and the five root registrations including controller/Herdr); the earlier shared-store incident is disclosed in rework-1381.md. Rehearsal now uses only a private Nix DB; no live root query was rerun.
|
||||
**Expected approved live Nix side effect:** each controller run makes three `nix-store --query --roots` calls, one each for old package, target package and loop, immediately after rooting those paths and before freeze/resume of offline work. **Every resume repeats all three queries and their possible pruning**, even if the roots already exist. Monitor/check/arm/continue/verify alone do not make those roots queries. Those queries can prune stale auto-root registrations and stale temporary-root files in Nix state, even though they do not delete store packages. Surma's go must include that side effect (and the five root registrations including controller/Herdr); the earlier shared-store incident is disclosed in rework-1381.md. Rehearsal now uses only a private Nix DB; no live root query was rerun.
|
||||
|
||||
## 4. Driver: started → launch in the same pane → fresh verifier
|
||||
|
||||
The driver checks the pinned shell still owns the selected pane and no unexpected foreground process exists. It checkpoints **`started` before any launch attempt**, then submits the permanent target-owner path once via `herdr pane run` to the manifest's pane (live **w7:p6**). No shell reconstruction or in-world mediator action is needed during downtime.
|
||||
|
||||
Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. The wait has **one absolute 120 s deadline**, from cron `src/types.rs` AddParams' existing registration/publication budget (also the earlier test_driver completion bound); it is not a periodic scan. A missing registration writes failure evidence, submits the Surma notification, and the controller exits. The `started` checkpoint remains: **do not resume, restore or launch again automatically**. Herdr/RPC calls also use that existing finite bound. Inspect the pane and native identity once, preserve new work, then arrange a narrow repair/new explicit verifier. `driver.py verify` is still allowed after human verification; no automatic retry occurs. Then await actual gateway RPC completion (`cred.status` locked and registration acknowledgement). **Herdr wait-output is internally polling and is not used by the final driver.** It was an unsuccessful preliminary approach only.
|
||||
Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. The wait has **one absolute 120 s deadline**, from cron `src/types.rs` AddParams' existing registration/publication budget (also the earlier test_driver completion bound); it is not a periodic scan. Herdr/RPC calls use that existing finite bound. Missing registration or a failed post-start `cred.status`/subscribe/send produces the disk-independent failure/death signal and exits, keeping **`started`**. Never resume, restore or launch again automatically. Herdr wait-output is internally polling and is not used.
|
||||
|
||||
### Forward completion after a post-start failure (Surma's independent terminal)
|
||||
|
||||
1. Read checkpoint/monitor and inspect Herdr pane/native identities once. If target is not actually running, preserve all new work and obtain a narrow forward repair plan; this mode will refuse and **does not launch it**. Do not relaunch blindly, restore an old snapshot or hand-edit the checkpoint. If only registration/publication failed but exactly one target is running, use:
|
||||
|
||||
```sh
|
||||
cd /home/surma/sau-384-cutover
|
||||
PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py continue operator.json
|
||||
```
|
||||
|
||||
2. `continue` exclusively locks, binds the unchanged manifest, inspects that exact shell/pane, exactly one target host for this world and all offline target plugin hashes. It never freezes, applies, restores or launches. It then performs the **skipped exact RPCs**, through operator.json's gateway: `cred.status {}` must report locked and is durably saved as `driver.json.locked_cred_status`/`locked_cred:true` **before Surma unlocks**; `session.subscribe` uses id `cutover:<continuation_id>`, session `cutover-readiness`, method `web.session_event`, best_effort true; `session.send` uses session default, the **same** continuation_id, kind follow-up and exact continuation_text. It saves the receipt and reaches `awaiting_verification`. A recorded locked-startup proof is reused if a later retry occurs after UI unlock; absent that proof, unlocking first is a discrepancy, not permission to fake the check.
|
||||
3. If a method/reply fails again, keep target work, inspect the error and fix only that cause; repeat the **same continue command**, not migration/start. Stable subscribe/message IDs make this safe even when send committed and only its reply was lost. Both started and awaiting_verification are accepted. Tests injected each RPC failure and committed-then-lost send acknowledgement; one continuation run-start/consumption remained. The real 120 s registration timeout also recovered through this command without a second owner.
|
||||
4. `verify` still requires awaiting_verification. Run continue first when phase is started; there is no direct started→verified override. The fresh mediator can then verify/ask for unlock/release as §5; if locked cred prevents SSH signing, Surma performs the same approved local external commands. Live SSH/new-cred signing is not claimed tested. No hand-edited checkpoint or new continuation UUID is needed.
|
||||
|
||||
Finally the driver publishes the manifest's **new explicit default verification message** with its stable UUID and records `awaiting_verification`. The manager, not the driver, runs it. This names the new mediator as verifier without claiming the cut default run was recovered. The test produced one run-start and mailbox-consumed pair for that message. Publication is not immediate model execution: in the last gate it waited **23.753 s to run-start / 24.530 s to consumption** behind initial reconciliation. Do not confuse that with publication latency or a completed startup scan.
|
||||
|
||||
@@ -124,7 +155,7 @@ No automatic relaunch, whole-world restore or reapply is allowed after `started`
|
||||
|
||||
## 5. New mediator: locked proof → Surma unlock → git → release web → delete tool
|
||||
|
||||
The explicit new verification run reads the manifest, `driver.json`, migration/loss/audit evidence and this runbook. Keep ingress closed until checks pass:
|
||||
The explicit new verification run reads this runbook and in-world ownership/cred state. While cred is locked, Surma checks the external manifest/checkpoint/migration evidence in his independent terminal; the new mediator can read it through approved SSH only after UI unlock if signing works. Surma can also perform the final local verify command, so no vault-key transfer or untested signer is required to finish. Keep ingress closed until checks pass:
|
||||
|
||||
- one target manager and one runner ownership lock per running session; no old owners, mail deliverer or old orchestrator service;
|
||||
- cred **locked**, no vault/key transfer; grants/requests/settings preserved. Interrupted callers reuse their existing stable credential-request keys, not automatic retries of secret-backed mutations;
|
||||
@@ -147,9 +178,9 @@ Interim compatibility: observed live Git **f089b56d** still speaks new cred's ex
|
||||
|
||||
Only now use **`orchestrator.release` for SAU-375** to release its approved commit. Inspect that method's current typed arguments; do not invent them. Verify published asset hashes and ingress. No worker publishes anything. The rework tested approved asset hashes in temporary output, **not an actual live release or physical browser**.
|
||||
|
||||
The Surma unlock above also enables the existing approved signing grant for the final external SSH verification/deletion command; it is not a second vault handover. Never ask for Surma's password or transfer the old vault key. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain `awaiting_verification`, keep git activation/ingress/release gated, and do not reconstruct or rerun the migration.
|
||||
The Surma unlock above is a prerequisite for using the existing approved signing grant for the final external SSH verification/deletion command (live signing interoperability is still unverified); it is not a second vault handover. Never ask for Surma's password or transfer the old vault key. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain `awaiting_verification`, keep git activation/ingress/release gated, and do not reconstruct or rerun the migration.
|
||||
|
||||
Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`, `git_activated_after_unlock`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command:
|
||||
Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`, `git_activated_after_unlock`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command below after target cred is unlocked and signing works; **Surma may instead run the identical command in his independent Citadel terminal**, avoiding any dependency on in-world SSH signing:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py verify operator.json verification.json 2>> tool-source-and-delete.log
|
||||
@@ -161,36 +192,38 @@ It marks the saved plan/driver verified, logs source hashes/source-as-log, and *
|
||||
|
||||
| Boundary | Exact recovery rule |
|
||||
|---|---|
|
||||
| Waiting for arm, including consumed-marker/save race | World untouched. Repeated arm is finite even with no reader. After the saved controller exits, `resume` accepts waiting_for_arm and the durable marker. Or run abort (untouched) after obtaining the external controller lock. |
|
||||
| Armed/captured/owner_stopped/frozen; no target attempt | External operator can use `... driver.py resume operator.json` **only after the saved controller has exited**. It detaches a replacement; same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed. |
|
||||
| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively Surma runs the real driver's **abort** mode below: it restores every approved changed path from the immutable complete backup, verifies the full frozen inventory, and launches the explicit old-owner in the selected pane. It never restores an incomplete backup. If planning never completed, apply cannot have started, so it verifies any saved inventory and relaunches the untouched old world. Restore is before/after-hash guarded, resumable, and writes main DBs before their old sidecars without opening them. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
|
||||
| `started` or later, even if boot seems failed | No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them and plan a narrow forward repair or coordinated revert. |
|
||||
| Waiting for arm, including consumed-marker/save race | World untouched. Repeated arm is finite even with no reader. After the saved controller exits, `resume` accepts waiting_for_arm and the durable marker; attach a new `monitor` for that replacement before it continues. Or run abort (untouched) after obtaining the external controller lock. |
|
||||
| Armed/captured/owner_stopped/frozen; no target attempt | External operator can use `... driver.py resume operator.json` **only after the saved controller has exited**. It detaches a replacement and returns; Surma then runs the same foreground monitor command. The replacement waits for its own monitor receipt before effects. Same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed. |
|
||||
| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively Surma runs the real driver's **abort** mode below: it restores every approved changed path from the immutable complete backup, verifies the full frozen inventory, and launches the explicit old-owner in the selected pane. It never restores an incomplete backup. If planning never completed, apply cannot have started: abort first discards only `migration/backup` (including its partial copy temporaries), retaining planning/inventory/log evidence, then checks space **before any checkpoint save**. If enough space is available, it verifies any saved inventory and relaunches the untouched old world. If a competing writer still fills the disk it refuses before writes/launch; Surma must free only identified, separately approved non-world data and repeat abort. Never delete a complete backup/plan or unrelated world data to make space. Restore is before/after-hash guarded, resumable, and writes main DBs before their old sidecars without opening them. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
|
||||
| `started` or later, even if boot seems failed | No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them. Use the forward-only continue command above for readiness/post-start RPC failure with one healthy target; otherwise plan a narrow forward repair or coordinated revert. |
|
||||
| Any plugin core.revert | Offline-swapped prev now means exactly pre-cut current (hash-tested), not pre-pre-cut. **Do not use blind individual core.revert across incompatible contracts.** Coordinate providers/callers and preserve new work; the session prerequisite below still applies. Later git activation uses normal core prev semantics. |
|
||||
| Session revert after target activity | Close gateway/ingress. While new manager is available, **unsubscribe web-chat**, verify removal, and prevent gateway re-registration while the old session binary runs. Never call web.start against that old manager or restore a registry backup that reintroduces it. Actual old-manager/gateway tests proved unsubscribe safe and ID-only re-key unsafe: the old binary ignores best_effort and replays history once. |
|
||||
| Host reboot / changed PID identity / unexpected survivor | Refuse saved native identities; re-inventory externally. Never signal a reused PID, blindly kill a group, or infer quiescence from a stop request. |
|
||||
|
||||
### Exact external failure commands (Surma, not an in-world mediator)
|
||||
|
||||
In the independent Citadel terminal, `cd /home/surma/sau-384-cutover`; read `driver.json`, `failure.json`, `driver.log` and the pane once. Commands use the pinned interpreter:
|
||||
In the independent Citadel terminal, `cd /home/surma/sau-384-cutover`; read `driver.json`, monitor output, any available `failure.json`/`driver.log`, and the pane once. At ENOSPC those disk-backed diagnostics may be absent or truncated; the terminal failure/death signal is authoritative that the controller ended, not that rollback is safe. Commands use the pinned interpreter:
|
||||
|
||||
```sh
|
||||
PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
|
||||
# Same manifest/boot/owners, known exited controller, pre-start only:
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py resume operator.json
|
||||
# After resume acknowledges, attach its new foreground monitor before effects:
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py monitor operator.json
|
||||
# Or abandon pre-start attempt, restore coherent old bytes, submit old-owner once:
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py abort operator.json
|
||||
```
|
||||
|
||||
Choose **one**, not both. Both use a nonblocking exclusive controller lock; do not delete the lock file or bypass it. Abort accepts waiting/armed/frozen/installed/restoring/restored only. It checkpoints `old_start_attempted` **before** pane launch, then `old_launched`. If interrupted during restoring/restored, repeat abort on the same manifest/checkpoints; if `old_start_attempted`/`old_launched`, inspect the pane instead of repeating it. Abort is forbidden after any target `started` attempt. Never call Kit.rollback by hand or relaunch a bare loop. If the pinned shell/pane changed, stop for explicit identity resolution; do not guess a replacement.
|
||||
|
||||
After abort, Surma verifies exactly one old package owner in the pane and old world readiness before retry. Preserve the frozen evidence, never delete it to make a check pass. For a clean retry **only after old_launched and verification**, archive checkpoints (not owners/selectors/GC roots/tool):
|
||||
After abort, Surma verifies exactly one old package owner in the pane and old world readiness before retry. Preserve the frozen evidence; the sole permitted deletion of backup evidence before verified success is the **reproducible incomplete pre-plan backup subtree** in the guarded abort path above. Complete backups are never deleted by abort. Every abort, including one with a completed plan, checks space before its first checkpoint save; a full disk is a deliberate pre-write refusal, not a failed checkpoint write. Abort rechecks the same measured space budget again immediately before old launch; if it refuses, free only identified/approved competing non-world data, not histories, jobs or complete backup evidence, then repeat abort on the same checkpoint. For a clean retry **only after old_launched and verification**, archive checkpoints (not owners/selectors/GC roots/tool):
|
||||
|
||||
```sh
|
||||
# Pick a fresh, unused archive path and record it; this example is this run.
|
||||
ARCHIVE=attempt-before-retry
|
||||
PYTHONDONTWRITEBYTECODE=1 "$PY" -B -c 'import json; assert json.load(open("driver.json"))["phase"] == "old_launched"'
|
||||
mkdir -- "$ARCHIVE"
|
||||
for p in driver.json freeze.json arm.json controller-process.json migration driver.log failure.json failure-notification.json preflight-live.json; do
|
||||
for p in driver.json freeze.json arm.json controller-process.json migration driver.log failure.json failure-notification.json preflight-live.json monitor.json poststart-inspection.json; do
|
||||
if [ -e "$p" ]; then mv -- "$p" "$ARCHIVE/"; fi
|
||||
done
|
||||
cp -- operator.json "$ARCHIVE/operator.json"
|
||||
|
||||
Reference in New Issue
Block a user