sausage/docs/SAU-384-rework-1392.md

SAU-384 round 4 — run 1392

Ready for review; not live execution approval. Addresses all five findings in extra-round-4.md. No shipped Rust/plugin behavior changed. Current-main source and state contracts were rechecked; all work was in the card tree, review artifacts and disposable box tc-zg0j56.

Selection and operator plan

The selected bundle is unchanged: cutover-targets-1389.tar.gz, 31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b, 23 payloads. All payload sizes/hashes and the native closure hashes were freshly checked. Host remains /nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0 from e09c4f9f. Cred remains 0db40101 / build 1636 (274 tests); mail/cron/rooms remain 1600/1602/1603. Deferred git remains approved 9cc9fa6f, activated only after target cred is live and Surma unlocks. No new WASM/package build. SDK-copy list: src/lib.rs only. Approved web 3d760267 remains held until the operator releases SAU-375 after session/web and verification prerequisites.

Exact commands and approval boundaries are in runbook.md. Current non-executable operator template: evidence/round4-1392/live-manifest.template.json. It still requires fresh caller/owner identities, guards and explicit Surma go. The only executable source is reconstructable as log, not retained as a tool. Surma is the external human recovery operator. The old mediator is deliberately cut; the driver publishes a new explicit default verification input, not inferred recovery.

Five findings fixed

1. Shared-filesystem ENOSPC. Space preflight measures POSIX allocated blocks for files/directories/root/non-followed links, and charges dense filesystem-fragment-rounded size where copy2 would expand a sparse file. It includes the concrete staging/prev/SDK/DB-copy roles; receipts save all measurements. Removed the false never-underestimates claim: metadata/normalization/growth/competing writers remain uncertainty. A separate sparse/tiny/directory fixture copied 4,599,808 allocated bytes from 12,288 source-allocated bytes, below its measured 4,603,904-byte budget. No live margin/default was invented.

   Mandatory foreground driver.py monitor receives memory-only Unix datagrams for inventory/backup progress and failure, and kernel pidfd controller-exit completion. Every start/resume waits for its own monitor; arm requires it. Failure JSON/log writes are best effort and cannot prevent CUTOVER FAILED or the independent Herdr notification attempt. A terminal still reports controller death on SIGKILL; no heartbeat/polling loop. Notification submission was tested, but returned shown:false/disabled; physical display is not claimed.

   Actual ENOSPC used one shared tmpfs for world, external root and driver.log, device 83, 0 free bytes, capacity 254,504,960 bytes derived from measured fixture/budget allocations. It failed during a real backup write after good preflight. Monitor showed failure and death despite unwritable disk evidence. Pre-plan abort removes only the reproducible incomplete backup subtree before any checkpoint write, retaining planning/inventory evidence. If the competing writer still fills the disk, it refuses before saving/launching. After that known test file was freed, the same abort restored coherent old ownership and readiness. Complete backups are never deleted. Every abort (with or without a completed plan) now checks space before its first checkpoint save; a second shared-filesystem zero-free-byte fixture proved the complete backup/checkpoint remained exact, then old relaunch worked after freeing only the known competing test file. Space is checked again before old relaunch. This is not proof against every filesystem's ENOSPC/power-loss behavior.

2. Post-start forward completion. New continue operator.json locks, checks the bound manifest, exact pane/shell, exactly one target host and target plugin hashes, and performs only missing/read-retry-safe post-start RPCs. It never launches/applies/restores. Actual locked cred.status is saved durably before unlock; subscribe uses the stable cutover registration; send reuses the exact continuation UUID/text. It reaches awaiting_verification, then normal verify/delete is reachable. Real failures of cred.status, subscribe, and send, including send committed before its acknowledgement was lost, all recovered with one run-start/consumption. A real 120 s absolute registration timeout recovered the same way without another owner. The value remains cron/src/types.rs's existing registration/publication bound. Runbook gives exact continue command, RPC payload derivation, locked-proof/unlock ordering and local Surma fallback if live SSH signing is unavailable.

3. Atomic temporary cleanup. A child was SIGKILLed inside atomic after writing a real partial plugin replacement temporary, before rename. Abort removes only a saved operation's newly created .cutover-tmp, absent in frozen inventory and prefix-matching an approved before/after image; unknown bytes/links refuse. Full frozen inventory equality passed inside rollback before old launch. Resume still overwrites/completes its own temporary. No arbitrary world cleanup.

4. Converter pre-freeze dry run. check/start, arm and immediate pre-freeze checks use the same pure cron converter as plan, on SQLite backup-API memory copies. interrupt:true, invalid text and non-object params refuse without freeze/mutation. Negative control: round-3 checker accepted the same invalid object payload, new checker refused. No compatibility code in a plugin.

5. Roots/wake-up documentation. Every controller resume repeats all three roots queries and their possible stale-registration pruning; this remains part of the proposed Surma-approved live plan. This round queried roots only against a private rehearsal Nix DB. No cron wake-up is needed: the stable explicit driver continuation owns handoff. A due-during-downtime job is skipped; a just-before-freeze input can enter an old pending queue that is intentionally dropped. Neither is replayed. The runbook disallows relying on guessed backup duration as readiness.

Testing also exposed a real Nix Python wrapper exec identity race in parent-side process capture. The launcher now receives the child's actual post-wrapper identity over a pipe-completion handshake. Controller kill/resume proved exact identity, a fresh monitor requirement and successful same-pane continuation. No liveness/endpoint polling was added.

Full logs and evidence

All log names below have prefix /logs/box/2026-10-31/tc-zg0j56-exec-. Evidence is evidence/round4-1392/.

Gate	Full stderr / outcome
Final shared world/root/log ENOSPC, pre-save incomplete-backup deletion, low-space refusal, old readiness; partial atomic SIGKILL/abort; all three post-start RPC failures; real registration timeout	166388571.stderr, clean combined pass on final tool source. Evidence under final/, including shared-enospc.json, atomic-abort.json, *-reviewed.json and registration-timeout.json.
Complete-plan ENOSPC on one shared filesystem: pre-write refusal, checkpoint/full-backup equality, old readiness after known competing-data cleanup	4094731865.stderr, pass; final/complete-plan-enospc.json.
Controller wrapper handshake, killed-controller pidfd, resume waits for fresh monitor, target completion	537908850.stderr, pass; resume-handshake.json, both monitor logs.
Converter pre-freeze refusals	1954198187.stderr, pass; dryrun.json.
Negative control + sparse allocation + all payload/native closure hashes + 133 final source/build inputs	1314968883.stderr, pass; converter-negative-control.json, allocated-space.json, source-and-bundle.json.
Committed WAL/hot rollback journal; exact pending/cancel/mail losses; preserved mail/jobs/IDs/calendar/outcomes/non-target JSON; apply interruption/rerun and physical rollback	2622539348.stderr, pass; sqlite-sidecars.json.
cargo test --workspace, unchanged final production source	3315145279.stderr, pass; 208 host integration tests, result at line 433, plus all workspace suites/doc tests.
Fresh target one-manager/locked-cred/original-byte preservation/pre-cut-prev verification; actual verify CLI and complete tool/test deletion	3030415406.stderr, pass; final/ final-ownership-preservation.json, verification.json, final-verified-driver.json, tool-deletion.json, test-deletion.json. Earlier verify/deletion pass: 1403220230.stderr.
In-memory source-log SHA/AST reconstruction; executables/bytecode still absent	2019074395.stderr, pass; final/source-log-roundtrip.json. Earlier docs/JSON/absence checks: 2041020851.stderr; final docs/JSON/absence/23-payload/133-source-input validation: 1011452644.stderr, final/final-validation.json.

Executed source deleted after verified success

Reconstruct only these three files from 3030415406.stderr for a later approved live operation; do not reconstruct tests:

• cutover_once.py: source lines 2–189, SHA-256 84b13644bd80c6b331dcd8df9bf79339f62fedb99f7dd376535b2bca7b0afd96;
• freeze.py: source lines 192–263, SHA-256 d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe;
• driver.py: source lines 266–669, SHA-256 5c3cba1a69b8b5e8f82ea83caa0697af494bf93dd0297f0c1b5231192cbb5fad.

After generalizing the no-write space gate to complete-plan and untouched abort too, the source was reconstructed for the final affected rerun, then the real verify command again marked that rehearsal verified, logged/deleted the three files and caches; all executed/reconstructed tests/patches/wrapper were logged/deleted too. Permanent owner/selector installation artifacts are distinct, not migration code. Live runbook still mandates deletion after verified success.

Failed approaches and limits

• First interpreter alias linked only bin/, losing Nix Python's site-package context. Linked the complete existing interpreter environment; imports and gates then passed.
• Cold workspace cache made six offline guest builds miss async-task; populated the disposable cache and reran the whole workspace, 208/208 host tests, unchanged source. Initial full log 2236066973.stderr, missing dependency at lines 755–785; final pass above. No retry-without-diagnosis claim.
• First shared filesystem test refused before freeze because actual source growth exceeded its provisioned measurement. Recorded that valid negative control, then provisioned the test from measured fixture/budget roles before inducing deliberate post-preflight pressure. A later test filled before the backup had copied any data: abort correctly needed a no-write space check before its first save; fixed and repeated with a before-backup event gate. Final shared test reached zero free bytes during copy and the final combined gate passed cleanly. Earlier fixture checkpoint-name collisions, an existing evidence-export directory and a busy mount affected cleanup/export only; used fresh identities/paths and explicit disposable-pane cleanup, never edited migration checkpoints. Mounts were left for disposable-box teardown, not live cleanup.
• First post-start fixture omitted session prompt files, so it correctly recorded run-end error rather than mailbox consumption. Added synthetic prompts and repeated all three failure modes from fresh worlds; each then produced exactly one consumed continuation.

This round's failure fixtures are small synthetic real-core/Herdr worlds, not another live-size benchmark. Prior accepted live-size race/replay/core-watch/upgrade/deadline/idle/grace/crash-supervision, scale and Git compatibility/unlocked activation evidence remain cumulative. Startup 0.6078 s, catch-up 0.0128 s, mail publication 0.5376 s vs callers' 3 s/10 s limits, catch-up outlier 12.0419 s and initial-scan delay remain disclosed in earlier reports; not remeasured or upgraded to guarantees here.

Unverified/out of scope: actual approved live cut-over/frozen caller audit, live detached SSH/Herdr w7:p6 execution, live SSH signing through target cred, Surma's real unlock/new-git external networking, physical notifications/browser, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. Surma's external terminal has exact continue/verify commands, so final completion is not dependent on the untested live signer. No push, live activation/restart/config/owner change, release, deployment or /www publication. Surma go still precedes mediator core/session/cred operation, the proposed detached-SSH exception/root side effects, and every push.