From 8bf2a3c661ead774689e508ed45f609b40f60dd1 Mon Sep 17 00:00:00 2001 From: SAU-384 implementer Date: Sat, 31 Oct 2026 13:31:06 +0000 Subject: [PATCH] Close handover race, upgrade readiness and gateway rollback rehearsal gaps Record real-core delay-gate tests and actual gateway rollback evidence; require unsubscribe rather than an unsafe ID-only re-key. Keep the external native installation gate explicit pending read-only inspection. Card: SAU-384 --- docs/SAU-384-rehearsal.md | 14 ++++++++----- docs/SAU-384-resume-1369.md | 39 +++++++++++++++++++++++++++++++++++++ docs/SAU-384-runbook.md | 8 ++++---- 3 files changed, 52 insertions(+), 9 deletions(-) create mode 100644 docs/SAU-384-resume-1369.md diff --git a/docs/SAU-384-rehearsal.md b/docs/SAU-384-rehearsal.md index a3dc5b2..ac3f7ac 100644 --- a/docs/SAU-384-rehearsal.md +++ b/docs/SAU-384-rehearsal.md @@ -7,10 +7,10 @@ Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover ## Required resolution before completing the kit 1. Pin the actual external host launcher/service owner, stop/start interface, host architecture and runtime closure. Those are not projected into this world. The supplied local native artifact is **aarch64-linux**, dynamically linked to the Nix paths printed in the build evidence; it is NOT a verified live-host package. Do not guess a systemd unit, use `core.restart` as stop-all, or install a mismatching binary. -2. Correct the rollback alternative: **unsubscribing works; merely re-keying the registration does not**. An actual pre-best-effort session component, reading a registration written by the target component, ignores `best_effort` and sends history to its new ID. Rehearsal: unsubscribe = one barrier callback and no gateway-shaped replay; re-key = barrier plus historical replay. The test used the actual old manager and a native callback sink, not browser SSE. Either make unsubscribe mandatory, or specify what stronger meaning of “re-key” is intended. Do not claim an ID rename is safe. -3. Complete remaining integrated acceptance: deterministic manager armed/exit-race injection on real core (the real callback-boundary sequence and primitive watch tests are not that injection), replacement orchestrator readiness after the observed upgrade, and old-manager/gateway rollback without re-registration. The old orchestrator's exit preceded the core's replacement service-start event; the immediate process snapshot still had no new owner, so readiness is not claimed. +2. Resolved rollback choice: **unsubscribing works; merely re-keying the registration does not**. An actual pre-best-effort session component, reading a registration written by the target component, ignores `best_effort` and sends history to its new ID. Rehearsal: unsubscribe = one barrier callback and no gateway-shaped replay; re-key = barrier plus historical replay. The test used the actual old manager and a native callback sink, not browser SSE. The runbook now makes unsubscribe mandatory, using the card's authorized option. Do not claim an ID rename is safe. +3. **Closed in run 1369:** armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed; actual old-manager/gateway rollback without re-registration reproduced unsubscribe safety and ID-only re-key replay. See [resume evidence](SAU-384-resume-1369.md). The delay/observation hooks were only in a disposable host copy, not the selected host artifact or committed code. -The safe unsubscribe option can be selected without inventing compatibility code. The runbook nevertheless withholds a live-ready claim until the card's rollback wording and host installation manifest are unambiguous. +Use the card's explicit **unsubscribe** rollback option; an ID rename is not a safe alternative. The remaining blocker is the actual external host launcher/native installation manifest. Read-only SSH credential request **055995a51608c57dd25b323bea731fcb** is pending; it grants no stop/restart permission. ## What was built and exercised @@ -30,11 +30,11 @@ The safe unsubscribe option can be selected without inventing compatibility code | Offline failure boundaries | Before-start rollback restored exact bytes; conflicting outside edits refused; ambiguous legacy `interrupt:true` refused without mutation. After-start apply and whole-world rollback both refused. | | Default/advisor HTTP replay | Three identical same-ID POSTs each, before/after consumption; exactly one answered run per session. Default returned null on repeats; advisor returned its same receipt. | | Explicit open target run | A run-start/consumed record with a finished-looking assistant tail, but no run-end, recovered under the **same agent-run UUID**, with one start. Legacy finished-looking history alone produced no target event/run. | -| Real VFS/runner-boundary work | Actual SDK `armed` then plugin-write `changed`; five follow-ups published at durable callback/runner-end boundaries produced five distinct runs, each consuming its one ID. Does not force the exact armed-exit seam. | +| Real VFS/runner-boundary work | Actual SDK `armed` then plugin-write `changed`; five callback-boundary runs. Run 1369 additionally forced publication before runner `armed` and after its empty-mailbox exit decision while detached completion was held; no lost/duplicate inputs (disposable host delay gates, actual target WASM). | | Real core detached errors | Full nested error `data` survived a detached `core.call`. | | Real cron research deadline | **305.427 s** send→stopped in the final gate; real cron invoked warning and Stop. The 305-second test deadline is the existing 300-second warning interval plus design grace 5 seconds, not a new default. Held inference was cut, no follow-up run. Original configured research default remains 1,200 seconds. | | Effective grace / crash supervision | Passing native host suite exercises real WASI uniform-grace and before/after-readiness crash-supervision fixtures. Deliberate `core.kill` is NOT a crash: it disables that boot service; an initial rehearsal expecting its restart failed, correctly, and was replaced by the existing crash tests. | -| Orchestrator upgrade/idle | Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU **0.02 s / 20.0002 s ≈ 0.10%**. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Replacement readiness remains unverified. | +| Orchestrator upgrade/idle | Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU **0.02 s / 20.0002 s ≈ 0.10%**. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Run 1369 additionally verified target acknowledgement, one owner and all 13 core watches armed after old exit. | ### Live-size latency after real records exist @@ -42,6 +42,10 @@ After producing actual run-start/run-end/mailbox-consumed records, subsequent fu The clean final live-size end-to-end gate measured manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, and `mail.send` publication to default **0.5376 s**. The integrated post-record mail sample was **0.5235 s**. Both publication samples are below the existing orchestrator **3 s** and cred **10 s** caller budgets (SAU-400 report cites the originating SAU-366 review). These warm-cache synthetic-loopback real-core observations are not live latency guarantees. First scanning still delays first new input after the fast boot acknowledgement. +## Resume verification + +Run 1369 forced recompilation and repeated the complete workspace suite on the restored original host source: pass, including **208 host integration tests**. All **144 source files** matched the selected source snapshot. Full log `/logs/box/2026-10-31/tc-asa3yr-exec-3838117964.stderr` (host tests lines 64–434; equivalence line 785). New tests and their limitations are detailed in `SAU-384-resume-1369.md`; new test controllers and instrumentation were deleted after successful checks. Original bundle and production source are unchanged. + ## Evidence and failures All following logs are under `/logs/box/2026-10-31/`: diff --git a/docs/SAU-384-resume-1369.md b/docs/SAU-384-resume-1369.md new file mode 100644 index 0000000..f9f0fe5 --- /dev/null +++ b/docs/SAU-384-resume-1369.md @@ -0,0 +1,39 @@ +# SAU-384 resume — run 1369 + +Authority remains the card, design Rollout and SAU-356 approval gates. No live cut-over has been authorized or executed. + +## Results added + +- **Armed race passed:** actual target session WASM on real core, held immediately after core registered the runner's watch but before `armed`; a second real `session.send` published during the hold. Both IDs were consumed once in one run. +- **Exit race passed:** held only delivery of the runner's initial result after the target's empty-mailbox exit decision. A second real publication arrived while the manager still considered that runner busy. Releasing detached completion produced exactly two runs, each consuming its own one ID. At publication the original runner had already exited; only the manager remained. No new timer/polling was added. +- **Orchestrator upgrade readiness passed:** actual old SAU-370 component → selected SAU-408 component through the native plugin file watcher, old exit at 13:23:42.625Z before target start at 13:23:42.747Z, successful target acknowledgement and exactly one new owner. All 13 paths in `service.rs::WATCHES` armed in the target process. Prior whole-host idle duty measurement remains ~0.10% over 20 s in the paused, empty-board/live-size fixture; not a live workload guarantee. +- **Actual gateway rollback passed:** new gateway first made its real `web-chat` best-effort registration; without another `web.start`, the session binary reverted to the actual pre-best-effort manager. Explicit unsubscribe caused zero `web.session_event` calls through the old manager's catch-up barrier. ID-only re-key caused one historical gateway callback. Therefore use the card's **unsubscribe** option; do not offer ID-only re-key as safe. This checks the actual callback/endpoint, not a physical browser/SSE display. +- Restored production host source after disposable observation/delay hooks, forced recompilation, and reran `cargo test --workspace`: pass, including 208 native host integration tests and grace/supervision tests. All 144 source files matched the selected source snapshot. No source behavior changed in any committed plugin or host. + +## Evidence + +Under `/docs/symphony/artifacts/SAU-384/evidence/`: +`resume-races-summary.json`, `resume-upgrade-summary.json`, `resume-upgrade-wire.json`, `resume-rollback-summary.json`, `resume-tool-deletion.json`, `resume-source-equivalence.json`. + +Full logs under `/logs/box/2026-10-31/`: +- `tc-asa3yr-exec-1457078935.stderr`: race passes, followed by the initial upgrade-observation failure. Immediate swap raced initial asynchronous graph discovery; no old-version discovery baseline existed yet. Next method waited once for discovery completion, not a liveness loop. +- `tc-asa3yr-exec-870753431.stderr`: successful upgrade readiness/order. +- `tc-asa3yr-exec-4006214778.stderr`: actual gateway unsubscribe/re-key comparison. +- `tc-asa3yr-exec-621695466.stderr`: executed tests/controller and host-only instrumentation as log, then deletion. No executable one-off/controller copies remain. +- `tc-asa3yr-exec-3838117964.stderr`: forced-recompile full workspace pass; host tests lines 64–434, source equivalence line 785. Earlier unforced repeat also passed but is not used to establish recompilation. + +The original migration tool remains deleted. Recreated only disposable test controllers to close the above gaps; deleted those too after verification. Retained target bundle hash/artifacts are unchanged. Source-as-log is evidence, not a retained executable migration kit. + +## Remaining work / approval wait + +Only the actual external launcher/native installation manifest is still unresolved. Source `/repos/nixenv/machines/citadel/default.nix:22–27` disables `services.sausage`; `/repos/nixenv/machines/nexus/service-sausage.nix:10–11` describes the current Herdr panel. These sources do not pin the live panel, world path, launcher command, restart owner, stop completion API, executable/runtime closure, or architecture. Do not substitute `systemctl stop sausage` or `core.restart`. + +Read-only systemd-web GETs confirmed no active system `sausage.service`, and user-manager journal entries mention `/home/surma/.config/herdr/herdr.sock`; they still do not identify the currently running panel. Sources used: +- `http://citadel-systemd.nexus.hosts.10.0.0.2.nip.io:8081/api/units?type=service` +- `http://citadel-systemd.nexus.hosts.10.0.0.2.nip.io:8081/api/logs?unit=user%401000.service&lines=100&boot=0` + +No attached grant existed. One credential request is pending: **055995a51608c57dd25b323bea731fcb**, `id_scout`, `sub:card:SAU-384`, card duration, for **read-only** Citadel launcher inspection. The initial attempt supplied the immutable item ID, but live request expects a name and rejected it before filing; retry with `id_scout` filed the sole request. Do not request again; reuse it on resume. A credential approval is not permission to stop/restart/deploy. + +After approval: inspect actual process/parent/Herdr configuration read-only; pin owner, world path, synchronous stop/start/completion mechanism and correct native package/closure. Finish/rehearse that specific outer host swap or report a precise unresolved interface. Do not silently claim that the existing target-host/native-controller rehearsal replaced the old native service package. The current local host bundle is an aarch64-linux debug-profile stripped binary, not a validated live installation package. + +The surviving throwaway box is `tc-asa3yr`; temporary build/state copies remain there until its TTL. It contains no executable migration controller. If it expires, restore from retained target bundle/evidence and read the source logs, not a shipped plugin compatibility path. No pushes, activations, restarts, releases, config/SDK mutation, or `/www` writes occurred in the live world. diff --git a/docs/SAU-384-runbook.md b/docs/SAU-384-runbook.md index 49b577f..5ae596b 100644 --- a/docs/SAU-384-runbook.md +++ b/docs/SAU-384-runbook.md @@ -1,6 +1,6 @@ # One-off handover operator runbook — blocked draft -Read [rehearsal/results](SAU-384-rehearsal.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes. +Read [rehearsal/results](SAU-384-rehearsal.md) and [resume verification](SAU-384-resume-1369.md), the [authority](/docs/design/plugin-handover/design.md) and [approval gates](/docs/symphony/artifacts/SAU-356/plan.md). This is **not** authorization to execute. Workers must not perform these live steps. Mediator/orchestrator alone lands and executes. ## 0. Finish the installation manifest (no live mutation) @@ -13,7 +13,7 @@ Bundle: `/docs/symphony/artifacts/SAU-384/target-artifacts.tar.gz`. - Inventory executable caller payloads in cron jobs, boot and other saved callback/action/configuration stores using their predecessor schemas. History text mentioning an old RPC is not an executable payload. The tested converter covers `jobs.data` where `method == session.send`; it does not claim a full live caller-store audit. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution. - Source references for predecessor formats: session `f9122585`, `src/main.rs` Paths (`pending.jsonl`, `cancel`) and `src/queue.rs`; cron `ba75c102`; target cron README's one-off conversion procedure. Active old component hashes are separately captured; do not equate a source reference with its compiled active binary without checking. - Confirm the frontend artifact is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`; keep `publish:hold` until step 5. Do not publish a worktree snapshot or release it early. -- Resolve the rollback rule before execution: an ID-only re-key is **unsafe**. Use explicit unsubscribe, and prevent the new gateway from re-registering `web-chat` against an old session binary. A different proposed “re-key” needs its own defined semantics/rehearsal. +- Rollback uses the card's explicit **unsubscribe** option. ID-only re-key is **unsafe**, proven again with the actual old manager and gateway in run 1369. Prevent the gateway from re-registering `web-chat` against an old session binary; never call `web.start` in that state. A different proposed “re-key” needs its own defined semantics/rehearsal. ## 1. Obtain approvals, then close ingress/ownership @@ -23,7 +23,7 @@ Run the one-off controller **outside the world and outside the service/process g Close the actual ingress and stop the **external host/service owner** synchronously using the pinned command. Await its shutdown/process-exit completion, not a loop over `/proc`, endpoint retry, sleep, or a set of stop requests. If an outer launcher would respawn the host, deliberately stop that owner too; do not edit boot config while agents can still run. `core.restart`/exit-75 is a restart request, not stop-all. The real core's `Host.stop` suppresses service restart admission and stops all trees in parallel. Design budgets are grace 5 seconds and separate EOF-stop 5 seconds, not arbitrary new timeouts. -The rehearsal's native external controller awaited `Host.stop`, asserted its process table was empty, then awaited host exit. The live equivalent needs the actual launcher identity from step 0. **That identity is currently missing; do not invent a unit name.** +The rehearsal's native external controller awaited `Host.stop`, asserted its process table was empty, then awaited host exit. The live equivalent needs the actual launcher identity from step 0. **That identity is currently missing; do not invent a unit name.** Source config has `services.sausage.enable = false` and describes a Herdr panel; read-only systemd-web inspection corroborates a user Herdr socket, not a system `sausage.service`. Credential request `055995a51608c57dd25b323bea731fcb` awaits approval for read-only inspection to pin the actual panel/launcher. It is not permission to stop or change it. ## 2. Back up frozen state and plan changes @@ -82,6 +82,6 @@ Verify the published approved asset hashes and ingress once, mark the migration | Before offline mutation | Stop with evidence; old frozen world is untouched. | | During swap, target never started | Resume the saved hash-guarded plan, or restore the full frozen backup and old external owner/configuration coherently. Do not partially start mixed contracts. | | Any target start attempted | No automatic full restore/reapply. New histories, mail, jobs, grants or external effects may already exist. Preserve new state; diagnose and plan a narrow forward repair/revert. Never blindly rerun an external effect or undo new work. | -| Session revert after target activity | A new approved operation: close gateway/ingress, `session.unsubscribe {id:"web-chat"}` while the new manager is still available, verify removal, prevent gateway re-registration while old session runs, then coordinate the stopped version change. Do not restore a registry backup that reintroduces the registration. ID-only rename is proven unsafe. Full actual gateway revert remains unverified. | +| Session revert after target activity | A new approved operation: close gateway/ingress, `session.unsubscribe {id:"web-chat"}` while the new manager is still available, verify removal, prevent gateway re-registration while old session runs, then coordinate the stopped version change. Do not restore a registry backup that reintroduces the registration. ID-only rename is proven unsafe. Run 1369 verified this with the actual gateway left running and no re-registration: zero historical callbacks after unsubscribe; ID-only re-key replayed history. Physical browser/SSE display is not claimed. | Independent plugin recovery follow-ups remain independent. Neither uniform grace nor this one-off migration proves their external side effects, usage accounting, crash recovery or provider cancellation are solved.