53 lines
11 KiB
Markdown
53 lines
11 KiB
Markdown
# SAU-384 review rework — run 1381
|
||
|
||
**Ready for review, not live execution.** This changes only local documentation and disposable one-off tooling. No shipped host/plugin behavior, live boot/SDK/launcher, activation, restart, publication, release or push changed. Surma's go still precedes the mediator's live operation and every push.
|
||
|
||
## Review findings addressed
|
||
|
||
1. **No mediator needed during downtime.** Replaced the disconnected procedures with one external CLI, `driver.py start → arm → detached run → verify`. `start` acknowledges a new OS session and blocks it on an arm FIFO. After arm the detached driver owns native freeze, immutable/resumable full backup, planning/frozen audit, offline apply, selectors, `started`, launch and readiness. It uses the exact pinned Herdr pane/shell, then publishes a **new explicit default verification message**, with a stable UUID, because the cut old mediator run is not recoverable. The last real-Core/Herdr test killed the invoking shell shim, then killed the detached controller at frozen, resumed it, and reached target in the same shell. Actual target run-start and mailbox-consumed records named that new verification message once. A real live SSH launch was not performed; the runbook makes the two finite foreground SSH launch/arm commands, and the required explicit go for that exception to SSH's background-work policy, part of the proposed operator plan.
|
||
2. **Actual frozen-state gates.** `Kit.plan`, apply, starting and pre-start rollback require the real `freeze.json`, phase frozen, current boot ID, matching world, dead pinned owners, empty old group, no subsequently running host, and unchanged checkpoint bytes. A caller-supplied stopped literal is gone. Captured/wrong-boot checkpoints, running owner, changed config/SQL projection and after-start resume refuse before mutation. Offline retries accept only their saved before/after images.
|
||
3. **Persistent installation pins.** Driver registers/verifies GC roots for exact old/target/loop packages and creates old/target selector and owner artifacts before freeze, so pre-start recovery already has the explicit old owner. Both wrappers export their own pinned selector. Cold restart in the same Herdr shell, without ambient SAUSAGE_FLAKE, ran the exact target; the explicit old wrapper ran the exact old package on a separate old-backup clone, never over new target work. Runbook prohibits bare loop/remote-default starts and explains that core.restart restarts the pin, not self-updates. Future host updates need an approved package/root/selector change; roots are not removed with migration scripts.
|
||
4. **Real PTY job control.** Used actual Herdr 0.9.0 and an interactive zsh job-control shell, not setsid+piped stdin alone. Loop was foreground process-group leader in the shell's session. Freeze sends TERM then CONT before pidfd wait. Separate injection left the real host in observed **T** state after outer-owner exit; TERM+CONT completed shutdown with no surviving group member. This exercises the stopped-state hazard; it does not claim every timing of naturally delivered SIGTTIN.
|
||
5. **Planning, SQLite and scale.** Incomplete backup resumes the same inventory/backup without cleanup or replacement. Streaming hashlib.file_digest avoids whole-file memory reads; os.walk/copy preserve directories and non-followed symlinks (including broken links). Complete DB sidecar families are backed up physically, recovered only in disposable copies, staged with SQLite backup in DELETE mode, and old sidecars removed before main-image replacement. Committed WAL and hot uncommitted rollback-journal fixtures passed interruption/rerun, exact losses, preserved job IDs/calendar/outcomes/generic JSON/unknown mail, and exact pre-start rollback. A full logical-size/count/bucket/directory-matched **synthetic** replica measured the whole-world backup path below; it was not an exact live-byte snapshot.
|
||
|
||
## Evidence and exact gates
|
||
|
||
Evidence directory: `/docs/symphony/artifacts/SAU-384/evidence/rework-1381/`. It contains only JSON manifests/inventories/checkpoints/results, not migration executables. `sandbox-operator-manifest.json` records the tested invocation; **do not execute it live**. `live-manifest.template.json` deliberately requires fresh identities, typed caller review/guards and a new stable continuation UUID before approval. Frozen guard tests used the real cron table projection; the live table/file set must include the board/config/registration review described in the runbook.
|
||
|
||
All logs below are `/logs/box/2026-10-31/`:
|
||
|
||
| Gate | Full stderr log / result |
|
||
|---|---|
|
||
| Final caller-killed + controller-killed/resumed + interrupted planning + same-Herdr-pane target chain | `tc-asa3yr-exec-775822946.stderr` — pass. 4,436 frozen regular files / 627,157,113 bytes in this session-size integration fixture; backup/hash stage 4.617 s. Earlier final chains also passed (`…-2599916298`, `…-2393656781`, `…-2876439273`). |
|
||
| Final process ownership/locked cred/new explicit verifier/preservation; stopped-PTY CONT; cold pinned target; explicit old package on old clone | `tc-asa3yr-exec-3942858286.stderr` — pass. `post-target-processes.json`, `post-preservation.json`, `post-continuation-records.json`, `post-pty-stop-continue.json`; only planned paths and enumerated new activity changed. |
|
||
| WAL + hot journal + interruption + exact rollback, repeated | `tc-asa3yr-exec-1595763713.stderr` — pass; first pass `…-1628589419`. `db-sidecars.json`. |
|
||
| Whole-world scale | `tc-asa3yr-exec-3620999999.stderr` — **138,444 files, 12,441,162,164 bytes, 45,310 directories**. Inventory **70.555 s**, copy plus double hash verification **135.385 s**, combined **205.940 s**. Exact functions used by Kit.plan; synthetic bytes/names, observed live size buckets/counts, warm-cache box. No cold-storage downtime SLA. |
|
||
| Current source equivalence + cargo test --workspace | `tc-asa3yr-exec-430706202.stderr` — pass; 121 Rust/Cargo inputs plus root Cargo/flake inputs matched current card tree. **208 host integration tests**, lines **64–431**, plus all workspace suites/doc tests. No Rust source changed or fresh plugin build needed. |
|
||
| Negative config/frozen caller controls, selected bundle rehash, verify and deletion | `tc-asa3yr-exec-3977808892.stderr` — pass; `negative-controls.json`, `bundle-recheck.json`, `tool-deletion.json`, `executed-test-deletion.json`. |
|
||
| Source-as-log reconstruction/hash/AST check; no scripts/bytecode recreated | `tc-asa3yr-exec-3884064136.stderr` — pass; `source-log-roundtrip.json`. |
|
||
|
||
### Reconstructable, but deleted after success
|
||
|
||
The executed tool was three small files, not a plugin or host patch. Exact source survives **as log only** in `tc-asa3yr-exec-3977808892.stderr`:
|
||
|
||
- `cutover_once.py`: lines **7–152**, SHA-256 `202eb7e635d675bb3960317913ae154e4f699fb7f5fe25a53ef34c965d0869c5`;
|
||
- `freeze.py`: lines **154–226**, SHA-256 `d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe`;
|
||
- `driver.py`: lines **228–393**, SHA-256 `7a26cfba9873541404f9661d5b0117f566301d408448cbb509a965d7900037aa`.
|
||
|
||
The verify CLI was actually executed, marked the rehearsal verified, logged/deleted these files, and removed caches. All executed test controllers were also logged/deleted; the named disposable Herdr server was stopped. Hash reconstruction was checked **in memory**, without recreating executable files. The live runbook mandates the same deletion. Permanent owner/selector artifacts are installation artifacts, represented as bytes/hashes in `permanent-launcher-artifacts.json`, not retained migration code.
|
||
|
||
## Selected builds unchanged
|
||
|
||
Selected bundle remains `/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`, SHA-256 `1bbcae4ef615ca8394b0c53b9a408441a49f8a618f059335c3bc248d958977c4`; all **22** payload hashes matched again. Host `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; SDK-copy **src/lib.rs only**. Mail/cred/cron/rooms builds **1600/1601/1602/1603** unchanged. Web artifact remains approved **3d760267**, held until target session/web verification and mediator/orchestrator release. No artifact repack or source change to a shipped plugin.
|
||
|
||
## Failures, corrections and limits
|
||
|
||
- Whole-tree transfer through box.put_tree rejected the 64 MiB service cap for repos/plugins/logs. Next method: read-only host size histogram, then regular-file size-bucket replica and the exact backup routines. This measures whole-world **scale**, not original live contents/path topology. The original accepted session byte snapshot still supplies the real-history integration volume.
|
||
- Initial PTY fixtures lacked state/core for console history, then matched an http-prefixed address against a bare address. Corrected fixture and barrier. Typeahead from two early console commands lost the second; final driver queues **one** subscription command. An early Herdr multiline output matcher stalled; inspection showed Herdr wait-output itself polls, so final code removed it entirely in favor of inotify completion. The actual registration path is `state/session/manager/registrations/subscriptions.json`, now corrected.
|
||
- **Shared-store foot gun:** an early default-store root query pruned **1,144 stale auto-root links and 571 stale temporary-root files** in shared Nix state. This was not an authorized live-config operation and no GC/package deletion was requested. Immediately stopped that method. Read-only SSH inspection on Citadel found **none** of those referenced targets and **none** of those PID numbers existing on the host, so no active roots required restoration. All subsequent root tests used a **private local Nix state database**, loaded from read-only closure metadata; no shared root query was repeated. Exact evidence/log: `gc-root-incident.json`, `tc-asa3yr-exec-2216542428.stderr`. This incidental stale-registration cleanup is disclosed rather than claiming literally zero host-state writes.
|
||
- One post-start assertion read history before the new verification message was consumed. Replaced it with arm-then-scan/history-write completion. In the final gate publication→run-start was **23.753 s**, →consumption **24.530 s**, queued behind startup reconciliation. This is **not** mail.send publication latency. Initial acknowledgement is not full scan completion.
|
||
|
||
Prior accepted replay/race/rollback/deadline/grace/crash-supervision/event-driven orchestrator evidence remains unchanged. Post-record manager ack **0.6078 s**, catch-up **0.0128 s**, mail publication **0.5376 s** / integrated **0.5235 s** remain below existing **3 s / 10 s** publication budgets; integrated catch-up outlier **12.0419 s**, ~0.10% idle duty, detached error propagation and **305.427 s** real cron warning/stop remain disclosed in implementation.md. This rework does not replace those measurements with a claim that the initial scan is fast.
|
||
|
||
Still unverified/out of scope: actual approved live cut-over/frozen backup and complete production caller guards, live SSH detach/Herdr w7:p6 execution, orchestrator.release/publication/physical browser, cold-storage timing, power loss, provider cancellation/billing and independent recovery follow-ups. The live launch exception needs explicit Surma go as part of the proposed plan; no such approval is claimed or requested by executing anything here.
|