Rehearse shared-disk failures and forward-only cutover completion

Card: SAU-384

PROTOCOL.md

@@ -19,7 +19,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
 | call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. |
 | interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. |
 | endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. |
-| service | A call in `/config/boot.yaml`. The core starts it at boot and again after an upgrade of its plugin. |
+| service | A call in `/config/boot.yaml`. The core starts it at boot, restarts it after a crash, and starts it again after an upgrade of its plugin. |
 | import | An interface of another plugin that a plugin uses. |
 | provider, consumer | The plugin that has an interface, and a plugin that imports it. |
 | journal | The central log of the world. |
@@ -70,7 +70,7 @@ Rust plugins use the SDK `sausage-plugin`. Its source is at
   `notes.rpc.discover` sends it `rpc.discover`.
 - `core.<method>` is a method of the core (section 9).
 - Method names that start with `rpc.` are reserved: `rpc.discover`,
-  `rpc.retired`, and `rpc.endpoint`.
+  `rpc.superseded`, `rpc.retired`, and `rpc.endpoint`.
 
 ### 3.3 What a call does
 
@@ -153,9 +153,17 @@ A process is an instance of one plugin version. When a new version becomes
 active (section 10.3):
 
 1. New calls get the new version at once.
-2. Each process of an older version gets a grace period to finish its work.
-   The default is 60 seconds. A plugin can declare its own value in
-   `rpc.discover` (`x-sausage-grace`). Then the core stops the process.
+2. At grace start, the core sends each process of an older version the
+   JSON-RPC notification `rpc.superseded {"current": "<sha256 of the new file>"}`
+   on stdin, without an ID. The process is still running: calls and responses
+   continue to work. It can stop accepting work, finish its in-flight calls,
+   and exit early. Ignoring the notice changes nothing. The grace period is
+   the upper bound before the core stops the process (section 3.6); its
+   duration is 5 seconds for every plugin. Discovery metadata, including an
+   explicit `x-sausage-grace`, does not change it. After grace, stdin closes
+   and the separate 5-second stop budget applies (section 3.6): at most two
+   5-second windows. Discovery-only processes do not receive this notice or
+   enter upgrade grace; they follow the discovery lifetime (section 10.1).
 3. When the old process of a service has ended, the core starts the service
    again with the new version.
 4. When the last process of an older version has ended, the core starts a
@@ -214,6 +222,7 @@ these areas fails.
 /proc/
   core/
     PROTOCOL.md          this file
+    discover.json        core.rpc.discover document
     graph.json           plugins, imports, and the result of each check
     graph.dot            the same graph for Graphviz
     locks.json           held locks and waiting processes
@@ -241,6 +250,40 @@ Rules:
 
 Principle: **inspect through files, act through calls.**
 
+### File watching
+
+`core.watch {"path": "/state/session/mailbox"}` stays open for the caller
+process's lifetime. It needs a request ID and a running plugin caller.
+Its first `core.notify` is `"armed"`, with empty params. Only then scan the
+files. Writes before arming are visible in that scan; writes after arming
+produce `"changed"` notifications (also with empty params). Rescan after
+each notification; files, not event counts, are the truth.
+
+Notifications follow successful plugin-originated file mutations: creation,
+truncation, descriptor/stream writes, rename (both source and destination),
+and deletion, including mounted `/run` and `/self` files. Paths are normalized
+in the virtual namespace; `/self` means the caller's `/proc/<pid>` folder.
+A watch matches itself and descendants, not sibling prefixes. Moving or
+removing an ancestor also invalidates it. Paths need not exist when armed.
+Open file writes follow plugin renames; unlinked/replaced handles no longer
+name visible files. External host writes are outside this contract.
+
+Changes may coalesce, including a change already visible in the initial
+scan, but a change after the last notification always produces another.
+Delivery holds no filesystem locks. Observation does not serialize descriptor
+IO; only overlapping opens and rename/removal wait for one another.
+Watch the narrowest useful path: delivery is not rate-limited, so watching a
+busy parent can cause a rescan for every write.
+
+The host-side watch ends only when its caller process exits, not on a change
+notification; there is no cancellation API or polling/retry loop. SDK callers
+should poll the watch future alongside their handler's work and drop it on
+return, not keep it in a background task that prevents an idle process from
+exiting. Dropping the future removes the SDK's pending entry and callback, so
+late notifications and responses are ignored; it does not cancel the host
+watch. A long-lived service keeps one watch per purpose for its lifetime,
+rather than repeatedly opening and dropping watches.
+
 ## 5. Endpoints
 
 An endpoint is a JSON-RPC interface that a running process serves over TCP,
@@ -317,10 +360,29 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
 - A service is a detached call with the parent `core`. The core waits for
   its result before it starts the next one. The result shows that the
   service is ready.
-- If a service fails, the core writes the error to the journal and continues.
-- After an upgrade of the plugin of a service, the core starts the service
-  again with the new version (section 3.7).
-- The core does not restart other detached processes.
+- If a service fails before answering, the core writes the error to the
+  journal and continues booting. A failure to admit a process (for example,
+  a missing plugin or invalid parameters) is logged, not retried in a loop.
+- The core supervises the process lifetime, not its readiness response. A
+  nonzero exit, trap or failed host task restarts the same service with its
+  original method and parameters, including a crash before readiness. A
+  readiness response alone never triggers a restart; a clean exit does not
+  count as a crash or trigger an immediate restart.
+- More than 5 crashes within 10 seconds gives up on that service and writes
+  `service gave up` to the journal. Each service has an independent sliding
+  window; crashes exactly 10 seconds old expire. There is no retry delay or
+  periodic liveness check. Giving up is latched for this host lifetime:
+  neither elapsed time nor an upgrade re-arms it.
+- After an upgrade of the plugin of a service, the same supervisor starts
+  it again with the current version after its old process ends (section
+  3.7). Superseded exits are retirement, not crashes. Admission and cleanup
+  reserve a single owner/attempt, so an upgrade racing a crash cannot start
+  two owners. A matching boot-service start from `rpc.retired` is ignored
+  (returns `null`), even while restart is pending or after giving up; older
+  plugins' fallback starts do not bypass supervision.
+- An explicit `core.stop` or `core.kill` disables automatic attempts for
+  that service, including during an upgrade. Host shutdown starts no new
+  processes. The core does not restart other detached processes.
 - A singleton is a service with a fixed endpoint in `/run`. Only the core
   starts it, so two instances cannot start at the same time.
 
@@ -328,7 +390,9 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
 
 | Method | Params | Result |
 | --- | --- | --- |
+| `core.rpc.discover` | none | OpenRPC core API document (also `/proc/core/discover.json`) |
 | `core.call` | `method`, `params`, `detach`? | the result of the method |
+| `core.watch` | `path` | stays open; `armed`, then `changed` notifications (section 4) |
 | `core.notify` | `name`, `params` | none (a notification) |
 | `core.stop` | `pid` | `{}` when the process has ended |
 | `core.kill` | `pid` | `{}` |
@@ -386,7 +450,8 @@ Params are by name. The document can have these extension fields:
   `methods`.
 - `x-sausage-imports`: for each provider, the part of its document that the
   plugin uses.
-- `x-sausage-grace`: the grace period in seconds.
+- `x-sausage-grace`: legacy grace declaration in seconds. The core ignores
+  this field; upgrade grace is always 5 seconds (section 3.7).
 
 The schema of params describes what the provider can read. The schema of a
 result describes what the provider writes. Descriptions are part of the

README.md

@@ -93,10 +93,16 @@ plugins/<name>/plugin.wasm   the current version of a plugin
 plugins/<name>/source/       its source
 state/<plugin>/              the state of a plugin, by convention
 state/core/journal.sqlite    the journal: stderr of all processes, and core events
-config/boot.yaml             the services that start with the world
+config/boot.yaml             ordered boot services, supervised for crashes (PROTOCOL §8)
 config/agent.yaml            the configuration of the agent
 ```
 
+Boot services restart on process crashes, not on readiness responses. More
+than 5 crashes within 10 seconds stops automatic attempts for that service
+and journals `service gave up`; an upgrade does not reset that decision.
+Explicitly stopping or killing a service also leaves it stopped for this
+host lifetime. See [Boot and services](PROTOCOL.md#8-boot-and-services).
+
 Rust builds use the SDK in `plugins/cargo/sdk/` directly. The host seeds this
 folder only when it is missing; restarts and host upgrades preserve existing
 files. SDK edits take effect on the next `cargo.build`, without a restart.

cli/src/activation.rs

@@ -279,15 +279,10 @@ impl Host {
                 .get(&candidate.plugin)
                 .filter(|old| !old.hash.is_empty() && old.hash != candidate.interface.hash)
             {
-                let grace = old.document["x-sausage-grace"]
-                    .as_f64()
-                    .filter(|value| value.is_finite() && *value >= 0.0)
-                    .unwrap_or(60.0);
                 upgrades.push(self.snapshot_upgrade(
                     &candidate.plugin,
                     old.hash.clone(),
                     &candidate.interface.hash,
-                    grace,
                 ));
             }
             self.0.journal.write(

cli/src/core.json

@@ -0,0 +1,19 @@
+{
+  "openrpc": "1.2.6",
+  "info": {"title":"core", "version":"0.1.0"},
+  "methods": [
+    {"name": "rpc.discover", "summary": "Describe the native core API.", "description": "Describe the native core API.", "paramStructure": "by-name", "params": [], "result": {"name": "result", "schema": {"type": "object"}}},
+    {"name": "watch", "summary": "Keep this call open until the caller process ends. First core.notify is armed, then changed after successful plugin-originated file mutations. Scan only after armed; rescan after changed. Events may coalesce; files are the truth. No polling or cancellation API. See PROTOCOL section 4.", "description": "Keep this call open until the caller process ends. First core.notify is armed, then changed after successful plugin-originated file mutations. Scan only after armed; rescan after changed. Events may coalesce; files are the truth. No polling or cancellation API. See PROTOCOL section 4.", "paramStructure": "by-name", "params": [{"name": "path", "required": true, "description": "Virtual file or directory path, including descendants; need not exist. /self is relative to the caller.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "null"}}},
+    {"name": "call", "summary": "Make a call and return its result; detach lets the new process outlive its caller.", "description": "Make a call and return its result; detach lets the new process outlive its caller.", "paramStructure": "by-name", "params": [{"name": "method", "required": true, "description": "Qualified method.", "schema": {"type": "string"}}, {"name": "params", "required": true, "description": "Named parameters.", "schema": {"type": "object"}}, {"name": "detach", "required": false, "description": "Detach the new process.", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {}}},
+    {"name": "notify", "summary": "Notify this process's caller while its call is open.", "description": "Notify this process's caller while its call is open.", "paramStructure": "by-name", "params": [{"name": "name", "required": true, "description": "Event name.", "schema": {"type": "string"}}, {"name": "params", "required": false, "description": "Event payload.", "schema": {}}], "result": {"name": "result", "schema": {"type": "null"}}},
+    {"name": "stop", "summary": "End a process and await cleanup.", "description": "End a process and await cleanup.", "paramStructure": "by-name", "params": [{"name": "pid", "required": true, "description": "Process ID.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "object"}}},
+    {"name": "kill", "summary": "End a process and await cleanup.", "description": "End a process and await cleanup.", "paramStructure": "by-name", "params": [{"name": "pid", "required": true, "description": "Process ID.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "object"}}},
+    {"name": "lock", "summary": "Acquire a FIFO process-lifetime lock.", "description": "Acquire a FIFO process-lifetime lock.", "paramStructure": "by-name", "params": [{"name": "path", "required": true, "description": "Lock key.", "schema": {"type": "string"}}, {"name": "wait", "required": false, "description": "Wait for acquisition (default true).", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {"type": "object", "properties": {"held": {"type": "boolean"}}, "required": ["held"]}}},
+    {"name": "unlock", "summary": "Release an owned lock.", "description": "Release an owned lock.", "paramStructure": "by-name", "params": [{"name": "path", "required": true, "description": "Lock key.", "schema": {"type": "string"}}], "result": {"name": "result", "schema": {"type": "null"}}},
+    {"name": "activate", "summary": "Install one or more checked plugin versions. See PROTOCOL section 10.", "description": "Install one or more checked plugin versions. See PROTOCOL section 10.", "paramStructure": "by-name", "params": [{"name": "plugin", "required": false, "description": "Single plugin name.", "schema": {"type": "string"}}, {"name": "wasm", "required": false, "description": "Candidate path.", "schema": {"type": "string"}}, {"name": "plugins", "required": false, "description": "Batch candidates.", "schema": {"type": "array", "items": {"type": "object", "properties": {"plugin": {"type": "string"}, "wasm": {"type": "string"}}, "required": ["plugin", "wasm"]}}}, {"name": "force", "required": false, "description": "Skip compatibility rejection.", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {"type": "object"}}},
+    {"name": "revert", "summary": "Restore the previous plugin version.", "description": "Restore the previous plugin version.", "paramStructure": "by-name", "params": [{"name": "plugin", "required": true, "description": "Plugin name.", "schema": {"type": "string"}}, {"name": "force", "required": false, "description": "Skip compatibility rejection.", "schema": {"type": "boolean"}}], "result": {"name": "result", "schema": {"type": "object"}}},
+    {"name": "log", "summary": "Read the central journal (PROTOCOL section 7).", "description": "Read the central journal (PROTOCOL section 7).", "paramStructure": "by-name", "params": [{"name": "pid", "required": false, "description": "Process ID.", "schema": {"type": "string"}}, {"name": "plugin", "required": false, "description": "Plugin name.", "schema": {"type": "string"}}, {"name": "since", "required": false, "description": "ISO 8601 start time.", "schema": {"type": "string"}}, {"name": "until", "required": false, "description": "ISO 8601 end time.", "schema": {"type": "string"}}, {"name": "grep", "required": false, "description": "Line substring.", "schema": {"type": "string"}}, {"name": "limit", "required": false, "description": "Maximum entries (default 100).", "schema": {"type": "integer"}}], "result": {"name": "result", "schema": {"type": "array", "items": {"type": "object"}}}},
+    {"name": "restart", "summary": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "description": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "paramStructure": "by-name", "params": [], "result": {"name": "result", "schema": {"type": "object"}}},
+    {"name": "exit", "summary": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "description": "Request clean host restart through the loop supervisor; needs explicit operator approval.", "paramStructure": "by-name", "params": [], "result": {"name": "result", "schema": {"type": "object"}}}
+  ]
+}

cli/src/core.rs

@@ -12,6 +12,8 @@ impl Host {
         p: Value,
     ) -> RpcResult {
         match method {
+            "rpc.discover" => Ok(serde_json::from_str(include_str!("core.json")).unwrap()),
+            "watch" => self.watch(caller, id, required_str(&p, "path")?).await,
             "call" => {
                 self.route(
                     caller.into(),

cli/src/filesystem.rs

@@ -47,27 +47,48 @@ impl Host {
         tracker: Process,
     ) -> anyhow::Result<Mounts> {
         let root = self.0.world.root.clone();
+        let operations = tracker.clone();
         let host = tracker
             .clone()
             .blocking(move || sausage_vfs::Host::new(root, tracker))
             .await?;
         let mut mounts = Mounts::default();
-        mounts.mount("/", Arc::new(host), false);
+        mounts.mount(
+            "/",
+            self.0
+                .observer
+                .layer(Arc::new(host), "/", operations.clone()),
+            false,
+        );
         mounts.mount("/proc", Arc::new(Proc(self.clone())), true);
         mounts.mount(
             "/self",
-            Arc::new(SelfFolder {
-                host: self.clone(),
-                pid,
-                memory,
-            }),
+            self.0.observer.layer(
+                Arc::new(SelfFolder {
+                    host: self.clone(),
+                    pid: pid.clone(),
+                    memory,
+                }),
+                &format!("/proc/{}", pid.as_deref().unwrap_or("console")),
+                operations.clone(),
+            ),
+            false,
+        );
+        mounts.mount(
+            "/run",
+            self.0
+                .observer
+                .layer(Arc::new(self.0.run.clone()), "/run", operations),
             false,
         );
-        mounts.mount("/run", Arc::new(self.0.run.clone()), false);
         Ok(mounts)
     }
     fn core_files(&self) -> Generated {
         let mut files = Generated::default();
+        files.insert(
+            "/discover.json",
+            || Ok(include_bytes!("core.json").to_vec()),
+        );
         let protocol = include_bytes!("../../PROTOCOL.md");
         files.insert("/PROTOCOL.md", move || Ok(protocol.to_vec()));
         let graph = self.graph_json();

cli/src/graph.rs

@@ -223,16 +223,7 @@ impl Host {
             if let Some(old) = old_plugins.get(plugin).filter(|old| {
                 !old.hash.is_empty() && old.hash != current.hash && current.error.is_none()
             }) {
-                let grace = old.document["x-sausage-grace"]
-                    .as_f64()
-                    .filter(|value| value.is_finite() && *value >= 0.0)
-                    .unwrap_or(60.0);
-                upgrades.push(self.snapshot_upgrade(
-                    plugin,
-                    old.hash.clone(),
-                    &current.hash,
-                    grace,
-                ));
+                upgrades.push(self.snapshot_upgrade(plugin, old.hash.clone(), &current.hash));
             }
         }
         self.publish_graph(plugins);

cli/src/host.rs

@@ -37,6 +37,8 @@ pub(crate) struct Inner {
     /// processes, and upgrade tasks do nothing.
     pub(crate) shutdown: std::sync::atomic::AtomicBool,
     pub(crate) run: sausage_vfs::Memory,
+    pub(crate) watches: crate::watches::Watches,
+    pub(crate) observer: sausage_vfs::observe::Observer,
     pub(crate) graph: Mutex<crate::graph::Graph>,
     pub(crate) activations: tokio::sync::Mutex<()>,
     pub(crate) watcher: Mutex<Option<notify::RecommendedWatcher>>,
@@ -91,6 +93,10 @@ impl Host {
         ))
     }
     fn with_parts(world: World, runtime: Arc<Runtime>, journal: journal::Journal) -> Self {
+        let watches = crate::watches::Watches::default();
+        let events = watches.clone();
+        let observer =
+            sausage_vfs::observe::Observer::new(move |path| crate::watches::changed(&events, path));
         Self(Arc::new(Inner {
             world,
             runtime,
@@ -109,6 +115,8 @@ impl Host {
             admission: tokio::sync::RwLock::new(()),
             shutdown: std::sync::atomic::AtomicBool::new(false),
             run: sausage_vfs::Memory::default(),
+            watches,
+            observer,
             graph: Mutex::new(crate::graph::Graph::default()),
             activations: tokio::sync::Mutex::new(()),
             watcher: Mutex::new(None),

cli/src/lib.rs

@@ -18,6 +18,7 @@ pub mod runtime;
 mod services;
 mod upgrades;
 mod watcher;
+mod watches;
 pub mod world;
 pub use host::Host;
 pub use native::{NativeContext, NativeFuture, NativePlugin};

cli/src/process.rs

@@ -33,6 +33,8 @@ pub(crate) struct Process {
     pub(crate) id: Option<Value>,
     pub(crate) answer: Option<oneshot::Sender<RpcResult>>,
     pub(crate) ended: watch::Sender<bool>,
+    pub(crate) stopped: bool,
+    pub(crate) superseded: bool,
 }
 impl Host {
     /// Called only after the execution task ends. Wait for the store and file operations before releasing locks.
@@ -57,6 +59,7 @@ impl Host {
                 .map(|_| self.0.journal.query_later(json!({"pid":pid,"limit":20})));
             self.0.journal.forget(pid);
             Self::release_locks(&mut state, pid);
+            self.service_ended(&mut state, pid, status, p.stopped, p.superseded);
             let status = status.to_string();
             tokio::spawn(async move {
                 if let Some(answer) = p.answer {
@@ -84,13 +87,24 @@ impl Host {
     }
     /// Close stdin and start the same five-second deadline for each attached child.
     pub(crate) fn begin_stop(&self, pid: &str) -> Option<watch::Receiver<bool>> {
+        self.stop_process(pid, false)
+    }
+    /// Grace expiry is retirement, not an operator stopping the service.
+    pub(crate) fn begin_retire(&self, pid: &str) -> Option<watch::Receiver<bool>> {
+        self.stop_process(pid, true)
+    }
+    fn stop_process(&self, pid: &str, retirement: bool) -> Option<watch::Receiver<bool>> {
         let mut state = self.0.state.lock().unwrap();
         let p = state.processes.get_mut(pid)?;
+        if !retirement {
+            p.superseded = false;
+        }
         let ended = p.ended.subscribe();
         if p.info.state == "stopping" {
             return Some(ended);
         }
         p.info.state = "stopping".into();
+        p.stopped = true;
         p.input.take();
         self.0.journal.write(pid, &p.info.plugin, "core", "stop");
         for lock in state.locks.values_mut() {
@@ -110,17 +124,24 @@ impl Host {
         let pid = pid.to_string();
         tokio::spawn(async move {
             tokio::time::sleep(std::time::Duration::from_secs(5)).await;
-            h.kill(&pid);
+            h.kill_process(&pid, retirement);
         });
         Some(ended)
     }
     /// Abort a process task and perform its ended-state cleanup.
     pub(crate) fn kill(&self, pid: &str) {
+        self.kill_process(pid, false);
+    }
+    fn kill_process(&self, pid: &str, retirement: bool) {
         let mut state = self.0.state.lock().unwrap();
         for lock in state.locks.values_mut() {
             lock.queue.retain(|(queued, _)| queued != pid);
         }
         if let Some(p) = state.processes.get_mut(pid) {
+            if !retirement {
+                p.superseded = false;
+            }
+            p.stopped = true;
             p.info.state = "stopping".into();
             p.input.take();
             self.0.journal.write(pid, &p.info.plugin, "core", "kill");

cli/src/routing.rs

@@ -51,15 +51,20 @@ impl Host {
             .processes
             .get(pid)
             .and_then(|p| p.answer.as_ref().map(|_| (p.caller.clone(), p.id.clone())));
-        if let Some((caller, id)) = target {
-            let notification = json!({
-                "jsonrpc": "2.0",
-                "method": "core.notify",
-                "params": { "call": id, "name": name, "params": params }
-            });
-            self.send(&caller, notification);
+        if let Some((caller, Some(id))) = target {
+            self.call_notify(&caller, &id, name, params);
         }
     }
+    /// Deliver a notification on an open core call using the same envelope as plugin progress.
+    pub(crate) fn call_notify(&self, caller: &str, id: &Value, name: &str, params: Value) {
+        self.send(
+            caller,
+            json!({
+                "jsonrpc":"2.0", "method":"core.notify",
+                "params": {"call":id, "name":name, "params":params}
+            }),
+        );
+    }
     /// Validate and route a request, then start its native or WASI process.
     pub(crate) fn route(
         &self,
@@ -166,6 +171,33 @@ impl Host {
                 {
                     return Err(error(-32600, "caller stopped or ended"));
                 }
+                // Older plugins may still try to start their boot service from
+                // rpc.retired. The supervisor owns it even while a restart is
+                // pending, or after it has given up. Do not create a second owner.
+                if state
+                    .processes
+                    .get(&caller)
+                    .is_some_and(|p| p.info.method == "rpc.retired")
+                    && state
+                        .services
+                        .iter()
+                        .any(|s| s.method == format!("{plugin}.{method}") && s.params == params)
+                {
+                    return Ok(Value::Null);
+                }
+                if caller == "core"
+                    && let Some(service) = state
+                        .services
+                        .iter_mut()
+                        .find(|s| Some(&s.request) == id.as_ref())
+                {
+                    if !service.enabled || service.owner.is_some() {
+                        return Err(error(-32600, "service attempt is no longer current"));
+                    }
+                    service.owner = Some(pid.clone());
+                    service.pending = false;
+                    service.version = Some(info.version.clone());
+                }
                 host.0
                     .journal
                     .write(&pid, plugin, "core", &format!("start {method} {params}"));
@@ -182,6 +214,8 @@ impl Host {
                         id: id.clone(),
                         answer: if id.is_some() { Some(tx) } else { None },
                         ended: watch::channel(false).0,
+                        stopped: false,
+                        superseded: false,
                     },
                 );
             }

cli/src/services.rs

@@ -1,12 +1,36 @@
-//! Ordered boot services and their identities for upgrade restarts (PROTOCOL.md section 8).
-use crate::Host;
+//! Ordered boot calls and the single restart owner for each service (PROTOCOL.md section 8).
+use crate::{Host, host::State};
 use serde_json::{Value, json};
-#[derive(Clone)]
+use std::{collections::VecDeque, sync::atomic::Ordering, time::Duration};
+use tokio::time::Instant;
+
+// Agreed systemd-derived values: /docs/design/plugin-handover/design.md,
+// https://www.man7.org/linux/man-pages/man5/systemd-system.conf.5.html.
+const CRASH_LIMIT: usize = 5;
+const CRASH_WINDOW: Duration = Duration::from_secs(10);
+
 pub(crate) struct Service {
     pub(crate) method: String,
     pub(crate) params: Value,
     pub(crate) request: Value,
-    pub(crate) starting: std::sync::Arc<std::sync::atomic::AtomicBool>,
+    pub(crate) owner: Option<String>,
+    pub(crate) pending: bool,
+    pub(crate) version: Option<String>,
+    pub(crate) enabled: bool,
+    crashes: VecDeque<Instant>,
+}
+impl Service {
+    fn crashed(&mut self, now: Instant) -> bool {
+        while self
+            .crashes
+            .front()
+            .is_some_and(|t| now.duration_since(*t) >= CRASH_WINDOW)
+        {
+            self.crashes.pop_front();
+        }
+        self.crashes.push_back(now);
+        self.crashes.len() <= CRASH_LIMIT
+    }
 }
 impl Host {
     /// Starts discovery in the background, then waits for each boot service in order.
@@ -24,30 +48,119 @@ impl Host {
         }
         Ok(())
     }
-    /// Records a unique request identity before the service can answer or exit.
+    /// Register once, before admission or readiness. Restarts reuse this registration.
     pub(crate) async fn start_service(&self, method: String, params: Value) {
-        if self.0.shutdown.load(std::sync::atomic::Ordering::SeqCst) {
-            return;
-        }
         let request = json!(uuid::Uuid::now_v7().to_string());
-        let starting = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(true));
-        self.0.state.lock().unwrap().services.push(Service {
-            method: method.clone(),
-            params: params.clone(),
-            request: request.clone(),
-            starting: starting.clone(),
-        });
+        {
+            let mut state = self.0.state.lock().unwrap();
+            if self.0.shutdown.load(Ordering::SeqCst) {
+                return;
+            }
+            state.services.push(Service {
+                method,
+                params,
+                request: request.clone(),
+                owner: None,
+                enabled: true,
+                pending: true,
+                version: None,
+                crashes: VecDeque::new(),
+            });
+        }
+        self.run_service(request).await;
+    }
+    /// Await readiness only; process cleanup, not this response, decides whether to restart.
+    async fn run_service(&self, request: Value) {
+        let (method, params) = {
+            let state = self.0.state.lock().unwrap();
+            if self.0.shutdown.load(Ordering::SeqCst) {
+                return;
+            }
+            let Some(service) = state
+                .services
+                .iter()
+                .find(|s| s.request == request && s.enabled)
+            else {
+                return;
+            };
+            (service.method.clone(), service.params.clone())
+        };
         self.0
             .journal
             .write("core", "core", "core", &format!("service start {method}"));
-        if let Err(error) = self
-            .route("core".into(), Some(request), method, params, true)
-            .await
+        let result = self
+            .route("core".into(), Some(request.clone()), method, params, true)
+            .await;
         {
+            let mut state = self.0.state.lock().unwrap();
+            if let Some(service) = state.services.iter_mut().find(|s| s.request == request) {
+                service.pending = false;
+            }
+        }
+        if let Err(error) = result {
+            // Admission failures have no process lifetime to supervise. A process crash
+            // before readiness is handled by finish just like one after readiness.
             self.0
                 .journal
                 .write("core", "core", "core", &format!("service failed {error}"));
         }
-        starting.store(false, std::sync::atomic::Ordering::SeqCst);
+    }
+    /// Called under the state lock after the old owner and its files/locks are gone.
+    pub(crate) fn service_ended(
+        &self,
+        state: &mut State,
+        pid: &str,
+        status: &str,
+        stopped: bool,
+        superseded: bool,
+    ) {
+        let Some(service) = state
+            .services
+            .iter_mut()
+            .find(|s| s.owner.as_deref() == Some(pid))
+        else {
+            return;
+        };
+        service.owner = None;
+        if self.0.shutdown.load(Ordering::SeqCst) || !service.enabled {
+            return;
+        }
+        if !superseded {
+            if stopped {
+                service.enabled = false;
+                return;
+            }
+            if status == "exit 0" {
+                return;
+            }
+            if !service.crashed(Instant::now()) {
+                service.enabled = false;
+                self.0.journal.write(
+                    "core",
+                    "core",
+                    "core",
+                    &format!(
+                        "service gave up {}: more than {CRASH_LIMIT} crashes within {} seconds",
+                        service.method,
+                        CRASH_WINDOW.as_secs(),
+                    ),
+                );
+                return;
+            }
+        }
+        self.queue_service(service);
+    }
+    /// Reserve the next attempt before spawning it; upgrades and crashes share this owner.
+    pub(crate) fn queue_service(&self, service: &mut Service) {
+        service.request = json!(uuid::Uuid::now_v7().to_string());
+        service.pending = true;
+        let request = service.request.clone();
+        let host = self.clone();
+        tokio::spawn(async move {
+            host.run_service(request).await;
+        });
     }
 }
+
+#[cfg(test)]
+mod tests;

cli/src/services/tests.rs

@@ -0,0 +1,356 @@
+//! Deterministic service supervision tests: native tasks stand in for guest lifetimes.
+use super::*;
+use crate::{NativePlugin, runtime::Runtime};
+use std::sync::Arc;
+use tokio::sync::{mpsc, oneshot, watch};
+
+struct Rig {
+    host: Host,
+    starts: mpsc::UnboundedReceiver<(String, oneshot::Sender<()>)>,
+    _world: tempfile::TempDir,
+}
+impl Rig {
+    async fn new() -> Self {
+        let world = tempfile::tempdir().unwrap();
+        let runtime = Runtime::new(world.path()).unwrap();
+        let host = Host::test_world(world.path(), runtime).await.unwrap();
+        let (tx, starts) = mpsc::unbounded_channel();
+        let plugin: NativePlugin = Arc::new(move |ctx, method, _| {
+            let tx = tx.clone();
+            Box::pin(async move {
+                if method != "serve" {
+                    return Ok(Value::Null);
+                }
+                let (end, ended) = oneshot::channel();
+                tx.send((ctx.pid, end)).unwrap();
+                let _ = ended.await;
+                Ok(Value::Null)
+            })
+        });
+        host.register_native("demo", plugin.clone());
+        host.register_native("other", plugin);
+        Self {
+            host,
+            starts,
+            _world: world,
+        }
+    }
+    async fn start(&mut self, method: &str) -> (String, oneshot::Sender<()>) {
+        let host = self.host.clone();
+        let method = method.to_string();
+        tokio::spawn(async move {
+            host.start_service(method, json!({"kept":true})).await;
+        });
+        self.next().await
+    }
+    async fn next(&mut self) -> (String, oneshot::Sender<()>) {
+        // Reuse the host integration call bound (activation.rs::call).
+        tokio::time::timeout(Duration::from_secs(60), self.starts.recv())
+            .await
+            .unwrap()
+            .unwrap()
+    }
+    fn ended(&self, pid: &str) -> watch::Receiver<bool> {
+        self.host.0.state.lock().unwrap().processes[pid]
+            .ended
+            .subscribe()
+    }
+    async fn crash(&self, pid: &str) {
+        let mut ended = self.ended(pid);
+        // Abort execution without asking the supervisor for an intentional stop.
+        self.host.0.state.lock().unwrap().processes[pid]
+            .abort
+            .as_ref()
+            .unwrap()
+            .abort();
+        ended.wait_for(|done| *done).await.unwrap();
+    }
+    async fn clean(&self, pid: &str, end: oneshot::Sender<()>) {
+        let mut ended = self.ended(pid);
+        end.send(()).unwrap();
+        ended.wait_for(|done| *done).await.unwrap();
+    }
+    fn assert_single(&self, pid: &str, registrations: usize) {
+        let state = self.host.0.state.lock().unwrap();
+        assert_eq!(state.services.len(), registrations);
+        let p = &state.processes[pid];
+        assert_eq!(p.info.params, json!({"kept":true}));
+        assert!(p.info.detached);
+        assert_eq!(p.caller, "core");
+        assert_eq!(
+            state
+                .services
+                .iter()
+                .filter(|s| s.owner.as_deref() == Some(pid))
+                .count(),
+            1
+        );
+    }
+    fn no_start(&mut self) {
+        assert!(self.starts.try_recv().is_err());
+    }
+}
+
+#[tokio::test(start_paused = true)]
+async fn crash_before_and_after_readiness_observes_process_not_response() {
+    let mut rig = Rig::new().await;
+    let (pid, _end) = rig.start("demo.serve").await;
+    rig.assert_single(&pid, 1);
+    rig.crash(&pid).await; // No ready RPC response was sent.
+    let (pid, end) = rig.next().await;
+    rig.assert_single(&pid, 1);
+    rig.host.answer(&pid, Ok(json!({"ready":true})));
+    tokio::task::yield_now().await;
+    rig.no_start(); // Ready does not end the service lifetime.
+    rig.crash(&pid).await;
+    drop(end);
+    let (pid, end) = rig.next().await;
+    rig.assert_single(&pid, 1);
+    rig.clean(&pid, end).await;
+    rig.no_start(); // A clean, intentional return is not a crash.
+    rig.host.stop().await;
+}
+
+#[tokio::test(start_paused = true)]
+async fn fifth_restarts_sixth_gives_up_and_journals_once() {
+    let mut rig = Rig::new().await;
+    let (mut pid, mut end) = rig.start("demo.serve").await;
+    for crash in 1..=CRASH_LIMIT {
+        rig.crash(&pid).await;
+        drop(end);
+        (pid, end) = rig.next().await;
+        rig.assert_single(&pid, 1);
+        assert_eq!(
+            rig.host.0.state.lock().unwrap().services[0].crashes.len(),
+            crash
+        );
+    }
+    rig.crash(&pid).await;
+    drop(end);
+    rig.no_start();
+    let logs = rig
+        .host
+        .call("core.log", json!({"grep":"service gave up"}))
+        .await
+        .unwrap();
+    assert_eq!(logs.as_array().unwrap().len(), 1);
+    assert!(
+        logs[0]["line"]
+            .as_str()
+            .unwrap()
+            .contains("more than 5 crashes within 10 seconds")
+    );
+    assert!(!rig.host.0.state.lock().unwrap().services[0].enabled);
+    // A retired hook cannot bypass give-up while its supervisor has no owner.
+    rig.host.register_native(
+        "fallback",
+        Arc::new(move |ctx, _, _| {
+            Box::pin(async move {
+                ctx.call(
+                    "core.call",
+                    json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
+                )
+                .await
+            })
+        }),
+    );
+    assert_eq!(
+        rig.host
+            .call("fallback.rpc.retired", json!({}))
+            .await
+            .unwrap(),
+        Value::Null
+    );
+    rig.no_start();
+    // Neither time passing nor an upgrade re-arms a given-up service.
+    tokio::time::advance(CRASH_WINDOW).await;
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
+    rig.host.begin_upgrade(upgrade);
+    tokio::task::yield_now().await;
+    rig.no_start();
+    rig.host.stop().await;
+}
+
+#[test]
+fn sliding_window_boundary_and_progress() {
+    let now = Instant::now();
+    let service = || Service {
+        method: "demo.serve".into(),
+        params: json!({}),
+        request: json!("request"),
+        owner: None,
+        enabled: true,
+        pending: false,
+        version: None,
+        crashes: VecDeque::new(),
+    };
+    let mut before = service();
+    let mut exact = service();
+    let mut after = service();
+    for _ in 0..CRASH_LIMIT {
+        assert!(before.crashed(now));
+        assert!(exact.crashed(now));
+        assert!(after.crashed(now));
+    }
+    assert!(!before.crashed(now + CRASH_WINDOW - Duration::from_nanos(1)));
+    assert!(exact.crashed(now + CRASH_WINDOW));
+    assert_eq!(exact.crashes.len(), 1);
+    assert!(after.crashed(now + CRASH_WINDOW + Duration::from_nanos(1)));
+    // Repeated progress beyond the window never accumulates old failures.
+    for step in 2..=CRASH_LIMIT as u32 + 1 {
+        assert!(exact.crashed(now + CRASH_WINDOW * step));
+        assert_eq!(exact.crashes.len(), 1);
+    }
+}
+
+#[tokio::test(start_paused = true)]
+async fn services_have_independent_crash_budgets() {
+    let mut rig = Rig::new().await;
+    let (mut a, mut a_end) = rig.start("demo.serve").await;
+    let (b, b_end) = rig.start("other.serve").await;
+    for _ in 0..CRASH_LIMIT {
+        rig.crash(&a).await;
+        drop(a_end);
+        (a, a_end) = rig.next().await;
+    }
+    rig.crash(&a).await;
+    drop(a_end);
+    rig.crash(&b).await;
+    drop(b_end);
+    let (b, b_end) = rig.next().await;
+    rig.assert_single(&b, 2);
+    rig.clean(&b, b_end).await;
+    rig.no_start();
+    rig.host.stop().await;
+}
+
+#[tokio::test(start_paused = true)]
+async fn upgrade_crash_race_has_one_restart_and_retired_fallback_cannot_duplicate_it() {
+    let mut rig = Rig::new().await;
+    let (pid, end) = rig.start("demo.serve").await;
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
+    rig.crash(&pid).await; // Exits between snapshot and begin_upgrade.
+    drop(end);
+    rig.host.begin_upgrade(upgrade);
+    let (pid, end) = rig.next().await;
+    rig.assert_single(&pid, 1);
+    assert!(
+        rig.host.0.state.lock().unwrap().services[0]
+            .crashes
+            .is_empty()
+    );
+    // A native retired hook exercises the same routing path as a guest fallback.
+    rig.host.register_native(
+        "fallback",
+        Arc::new(move |ctx, _, _| {
+            Box::pin(async move {
+                ctx.call(
+                    "core.call",
+                    json!({"method":"demo.serve","params":{"kept":true},"detach":true}),
+                )
+                .await
+            })
+        }),
+    );
+    assert_eq!(
+        rig.host
+            .call("fallback.rpc.retired", json!({}))
+            .await
+            .unwrap(),
+        Value::Null
+    );
+    rig.no_start();
+    rig.clean(&pid, end).await;
+    rig.host.stop().await;
+}
+
+#[tokio::test(start_paused = true)]
+async fn crash_before_upgrade_snapshot_also_has_one_restart() {
+    let mut rig = Rig::new().await;
+    let (pid, end) = rig.start("demo.serve").await;
+    rig.crash(&pid).await;
+    drop(end);
+    // Restart may be pending admission or already admitted: snapshot handles both.
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
+    rig.host.begin_upgrade(upgrade);
+    let (pid, end) = rig.next().await;
+    rig.assert_single(&pid, 1);
+    rig.no_start();
+    rig.clean(&pid, end).await;
+    rig.host.stop().await;
+}
+
+#[tokio::test(start_paused = true)]
+async fn pending_admission_is_single_owned_across_upgrade() {
+    let mut rig = Rig::new().await;
+    let (pid, end) = rig.start("demo.serve").await;
+    let host = rig.host.clone();
+    let admission = host.0.admission.write().await;
+    rig.crash(&pid).await;
+    drop(end);
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "native");
+    rig.host.begin_upgrade(upgrade);
+    assert!(rig.host.0.state.lock().unwrap().services[0].pending);
+    drop(admission);
+    let (pid, end) = rig.next().await;
+    rig.assert_single(&pid, 1);
+    rig.no_start();
+    rig.clean(&pid, end).await;
+    rig.host.stop().await;
+}
+
+#[tokio::test(start_paused = true)]
+async fn shutdown_rejects_restart_already_waiting_for_admission() {
+    let mut rig = Rig::new().await;
+    let (pid, end) = rig.start("demo.serve").await;
+    let host = rig.host.clone();
+    let admission = host.0.admission.write().await;
+    rig.crash(&pid).await;
+    drop(end);
+    rig.host.stop().await;
+    drop(admission);
+    tokio::task::yield_now().await;
+    rig.no_start();
+    assert!(rig.host.processes().is_empty());
+}
+
+#[tokio::test(start_paused = true)]
+async fn clean_service_stays_dormant_until_upgrade_and_explicit_stop_stays_disabled() {
+    let mut rig = Rig::new().await;
+    let (pid, end) = rig.start("demo.serve").await;
+    rig.clean(&pid, end).await;
+    rig.no_start();
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
+    rig.host.begin_upgrade(upgrade);
+    let (pid, end) = rig.next().await;
+    rig.assert_single(&pid, 1);
+    // An explicit stop stays disabled even if the task subsequently exits cleanly.
+    let mut ended = rig.host.begin_stop(&pid).unwrap();
+    end.send(()).unwrap();
+    ended.wait_for(|done| *done).await.unwrap();
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "newer");
+    rig.host.begin_upgrade(upgrade);
+    tokio::task::yield_now().await;
+    rig.no_start();
+    rig.host.stop().await;
+}
+
+#[tokio::test(start_paused = true)]
+async fn intentional_stop_and_shutdown_never_restart_even_during_upgrade() {
+    let mut rig = Rig::new().await;
+    let (pid, end) = rig.start("demo.serve").await;
+    let upgrade = rig.host.snapshot_upgrade("demo", "old".into(), "new");
+    // An explicit kill after supersession overrides the automatic upgrade restart.
+    rig.host
+        .call("core.kill", json!({"pid":pid}))
+        .await
+        .unwrap();
+    drop(end);
+    rig.host.begin_upgrade(upgrade);
+    tokio::task::yield_now().await;
+    rig.no_start();
+    let (_pid, end) = rig.start("other.serve").await;
+    rig.host.stop().await;
+    drop(end);
+    rig.no_start();
+}

cli/src/upgrades.rs

@@ -1,9 +1,15 @@
 //! Grace periods, service restarts, and retirement of replaced versions (PROTOCOL.md section 3.7).
-use crate::{Host, services::Service};
+use crate::Host;
 use serde_json::json;
 use std::{collections::BTreeMap, sync::atomic::Ordering, time::Duration};
 use tokio::sync::watch;
 
+/// Uniform host policy (PROTOCOL 3.7), independent of discovery metadata.
+const UPGRADE_GRACE: Duration = Duration::from_secs(5);
+
+#[cfg(test)]
+mod tests;
+
 pub(crate) struct OldProcess {
     pid: String,
     version: String,
@@ -12,9 +18,8 @@ pub(crate) struct OldProcess {
 pub(crate) struct Upgrade {
     plugin: String,
     previous: String,
-    grace: Duration,
+    current: String,
     processes: Vec<OldProcess>,
-    services: Vec<(Service, Option<watch::Receiver<bool>>)>,
 }
 impl Host {
     /// Snapshots old admissions while activation holds the admission write guard.
@@ -23,57 +28,59 @@ impl Host {
         plugin: &str,
         previous: String,
         current: &str,
-        grace: f64,
     ) -> Upgrade {
         let mut state = self.0.state.lock().unwrap();
         let processes = state
             .processes
-            .iter()
+            .iter_mut()
             .filter(|(_, process)| {
                 process.info.plugin == plugin
                     && process.info.version != current
                     && process.info.method != "rpc.discover"
             })
-            .map(|(pid, process)| OldProcess {
-                pid: pid.clone(),
-                version: process.info.version.clone(),
-                ended: process.ended.subscribe(),
+            .map(|(pid, process)| {
+                // The service supervisor owns the restart, including an exit on
+                // the superseded notice before the grace timer is installed.
+                process.superseded = !process.stopped;
+                OldProcess {
+                    pid: pid.clone(),
+                    version: process.info.version.clone(),
+                    ended: process.ended.subscribe(),
+                }
             })
             .collect();
-        let mut services = Vec::new();
-        let mut retained = Vec::new();
-        for service in std::mem::take(&mut state.services) {
-            if service
-                .method
-                .split_once('.')
-                .is_some_and(|(name, _)| name == plugin)
+        // A cleanly returned service is dormant until an upgrade. A pending
+        // admission will choose the current version itself. Disabled services
+        // (explicit stop or crash-limit give-up) must never be re-armed here.
+        for service in &mut state.services {
+            if service.enabled
+                && !service.pending
+                && service.owner.is_none()
+                && service.version.as_deref() != Some(current)
+                && service
+                    .method
+                    .split_once('.')
+                    .is_some_and(|(name, _)| name == plugin)
             {
-                let process = state
-                    .processes
-                    .values()
-                    .find(|process| process.id.as_ref() == Some(&service.request));
-                if process.is_some_and(|process| process.info.version == current)
-                    || (process.is_none() && service.starting.load(Ordering::SeqCst))
-                {
-                    retained.push(service);
-                } else {
-                    services.push((service, process.map(|process| process.ended.subscribe())));
-                }
-            } else {
-                retained.push(service);
+                self.queue_service(service);
             }
         }
-        state.services = retained;
         Upgrade {
             plugin: plugin.into(),
             previous,
-            grace: Duration::from_secs_f64(grace),
+            current: current.into(),
             processes,
-            services,
         }
     }
     /// Lets old calls finish, stops them after grace, and starts each lifecycle hook once.
     pub(crate) fn begin_upgrade(&self, upgrade: Upgrade) {
+        // Queue the notice before returning from activation, without waiting on guest code.
+        for process in &upgrade.processes {
+            self.send(
+                &process.pid,
+                json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":upgrade.current}}),
+            );
+        }
         let host = self.clone();
         tokio::spawn(async move {
             if host.0.shutdown.load(Ordering::SeqCst) {
@@ -83,7 +90,7 @@ impl Host {
                 "core",
                 &upgrade.plugin,
                 "core",
-                &format!("upgrade grace {} seconds", upgrade.grace.as_secs_f64()),
+                &format!("upgrade grace {} seconds", UPGRADE_GRACE.as_secs()),
             );
             let mut versions: BTreeMap<String, Vec<watch::Receiver<bool>>> = BTreeMap::new();
             versions.entry(upgrade.previous).or_default();
@@ -93,23 +100,11 @@ impl Host {
                     .or_default()
                     .push(process.ended.clone());
                 let host = host.clone();
-                let grace = upgrade.grace;
                 tokio::spawn(async move {
                     let mut ended = process.ended;
                     tokio::select! {
                         _ = ended.wait_for(|ended|*ended) => (),
-                        _ = tokio::time::sleep(grace) => { host.begin_stop(&process.pid); }
-                    }
-                });
-            }
-            for (service, ended) in upgrade.services {
-                let host = host.clone();
-                tokio::spawn(async move {
-                    if let Some(mut ended) = ended {
-                        let _ = ended.wait_for(|ended| *ended).await;
-                    }
-                    if !host.0.shutdown.load(Ordering::SeqCst) {
-                        host.start_service(service.method, service.params).await;
+                        _ = tokio::time::sleep(UPGRADE_GRACE) => { host.begin_retire(&process.pid); }
                     }
                 });
             }

cli/src/upgrades/tests.rs

@@ -0,0 +1,115 @@
+//! Controlled-clock coverage of both upgrade windows and discovery exclusion.
+use super::*;
+use crate::{
+    process::{Process, ProcessInfo},
+    runtime::Runtime,
+    world::World,
+};
+use tokio::{
+    sync::mpsc,
+    task::JoinHandle,
+    time::{Instant, advance},
+};
+
+/// A process with active work that ignores both superseded and EOF.
+fn process(
+    host: &Host,
+    pid: &str,
+    method: &str,
+) -> (mpsc::UnboundedReceiver<serde_json::Value>, JoinHandle<()>) {
+    let (input, incoming) = mpsc::unbounded_channel();
+    let task = tokio::spawn(std::future::pending());
+    host.0.state.lock().unwrap().processes.insert(
+        pid.into(),
+        Process {
+            info: ProcessInfo {
+                plugin: "notes".into(),
+                version: "old".into(),
+                method: method.into(),
+                params: json!({}),
+                parent: None,
+                detached: true,
+                start_time: 0,
+                state: "running".into(),
+            },
+            memory: sausage_vfs::Memory::default(),
+            files: sausage_vfs::Process::default(),
+            store_ended: watch::channel(true).0,
+            input: Some(input),
+            abort: Some(task.abort_handle()),
+            caller: "console".into(),
+            id: None,
+            answer: None,
+            ended: watch::channel(false).0,
+            stopped: false,
+            superseded: false,
+        },
+    );
+    (incoming, task)
+}
+
+/// PROTOCOL 3.7/3.6: five seconds running, then five seconds stopping; discovery is untouched.
+#[tokio::test]
+async fn two_five_second_windows_and_discovery_exclusion() {
+    let dir = tempfile::tempdir().unwrap();
+    let root = dir.path().join("world");
+    World::create(&root).unwrap();
+    let host = Host::test_world(&root, Runtime::new(&dir.path().join("cache")).unwrap())
+        .await
+        .unwrap();
+    let (mut input, work) = process(&host, "old-work", "work");
+    let (mut discovery_input, discovery) = process(&host, "old-discovery", "rpc.discover");
+    let upgrade = host.snapshot_upgrade("notes", "old".into(), "new");
+    assert_eq!(upgrade.processes.len(), 1);
+    tokio::time::pause();
+    let start = Instant::now();
+    host.begin_upgrade(upgrade);
+    assert_eq!(
+        input.recv().await.unwrap(),
+        json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new"}})
+    );
+    // Run the upgrade dispatcher, then its per-process task, so the deadline is armed.
+    tokio::task::yield_now().await;
+    tokio::task::yield_now().await;
+    let window = Duration::from_secs(5);
+    assert_eq!(UPGRADE_GRACE, window);
+    // tokio::time::sleep documents millisecond granularity; the paused driver
+    // may round a deadline up by one tick. Check each window independently.
+    let tick = Duration::from_millis(1);
+    let before_deadline = window - tick;
+    advance(before_deadline).await;
+    tokio::task::yield_now().await;
+    assert_eq!(host.processes()["old-work"].state, "running");
+    assert!(!work.is_finished());
+    advance(tick).await;
+    assert!(
+        input.recv().await.is_none(),
+        "EOF at grace, even with active work"
+    );
+    let stopped = Instant::now();
+    assert!((window..=window + tick).contains(&(stopped - start)));
+    assert_eq!(host.processes()["old-work"].state, "stopping");
+    // begin_retire schedules the independent PROTOCOL 3.6 deadline.
+    tokio::task::yield_now().await;
+    advance(before_deadline).await;
+    tokio::task::yield_now().await;
+    assert!(
+        !work.is_finished(),
+        "on_stop still has its full five seconds"
+    );
+    advance(tick).await;
+    assert!(work.await.unwrap_err().is_cancelled());
+    assert!((window..=window + tick).contains(&(Instant::now() - stopped)));
+    assert_eq!(host.processes()["old-discovery"].state, "running");
+    assert!(matches!(
+        discovery_input.try_recv(),
+        Err(mpsc::error::TryRecvError::Empty)
+    ));
+    assert!(!discovery.is_finished());
+    discovery.abort();
+    let _ = discovery.await;
+    // Synthetic processes have no execution cleanup task; discard only this test table.
+    host.0.shutdown.store(true, Ordering::SeqCst);
+    host.0.state.lock().unwrap().processes.clear();
+    host.stop().await;
+}

cli/src/watches.rs

@@ -0,0 +1,178 @@
+//! Armed plugin-file watches. A Notify permit coalesces changes without losing a later change.
+use crate::{Host, RpcResult, error};
+use sausage_vfs::observe::under;
+use serde_json::{Value, json};
+use std::{
+    collections::HashMap,
+    sync::{Arc, Mutex},
+};
+use tokio::sync::Notify;
+use uuid::Uuid;
+
+pub(crate) struct Watch {
+    path: String,
+    changed: Arc<Notify>,
+}
+pub(crate) type Watches = Arc<Mutex<HashMap<Uuid, Watch>>>;
+pub(crate) fn changed(watches: &Watches, path: &str) {
+    let targets: Vec<_> = watches
+        .lock()
+        .unwrap()
+        .values()
+        // Removing/moving an ancestor also invalidates a watch inside that subtree.
+        .filter(|w| under(path, &w.path) || under(&w.path, path))
+        .map(|w| w.changed.clone())
+        .collect();
+    for changed in targets {
+        changed.notify_one();
+    }
+}
+struct Registration {
+    watches: Watches,
+    key: Uuid,
+}
+impl Drop for Registration {
+    fn drop(&mut self) {
+        self.watches.lock().unwrap().remove(&self.key);
+    }
+}
+fn path(caller: &str, path: &str) -> String {
+    let path = sausage_vfs::resolve("/", path);
+    let path = path.trim_end_matches('/');
+    if path.is_empty() {
+        return "/".into();
+    }
+    if under(path, "/self") {
+        format!("/proc/{caller}{}", &path[5..])
+    } else {
+        path.into()
+    }
+}
+impl Host {
+    pub(crate) async fn watch(&self, caller: &str, id: Option<Value>, target: &str) -> RpcResult {
+        let id = id.ok_or_else(|| error(-32600, "watch needs a request ID"))?;
+        let mut ended = self
+            .0
+            .state
+            .lock()
+            .unwrap()
+            .processes
+            .get(caller)
+            .filter(|p| p.info.state == "running")
+            .map(|p| p.ended.subscribe())
+            .ok_or_else(|| error(-32600, "watch needs a running caller process"))?;
+        let changed = Arc::new(Notify::new());
+        let registration = Registration {
+            watches: self.0.watches.clone(),
+            key: Uuid::now_v7(),
+        };
+        self.0.watches.lock().unwrap().insert(
+            registration.key,
+            Watch {
+                path: path(caller, target),
+                changed: changed.clone(),
+            },
+        );
+        self.call_notify(caller, &id, "armed", json!({}));
+        loop {
+            tokio::select! {
+                biased;
+                _ = ended.wait_for(|ended| *ended) => return Ok(Value::Null),
+                _ = changed.notified() => self.call_notify(caller, &id, "changed", json!({})),
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    /// Pending changes coalesce; a change after consuming the last permit is retained.
+    #[tokio::test]
+    async fn coalescing_and_cleanup() {
+        let watches = Watches::default();
+        let n = Arc::new(Notify::new());
+        let r = Registration {
+            watches: watches.clone(),
+            key: Uuid::now_v7(),
+        };
+        watches.lock().unwrap().insert(
+            r.key,
+            Watch {
+                path: "/mailbox".into(),
+                changed: n.clone(),
+            },
+        );
+        changed(&watches, "/mailboxes/a");
+        assert!(
+            tokio::time::timeout(std::time::Duration::ZERO, n.notified())
+                .await
+                .is_err()
+        );
+        changed(&watches, "/mailbox/a");
+        changed(&watches, "/mailbox/b");
+        n.notified().await;
+        assert!(
+            tokio::time::timeout(std::time::Duration::ZERO, n.notified())
+                .await
+                .is_err()
+        );
+        changed(&watches, "/mailbox/c");
+        n.notified().await;
+        drop(r);
+        assert!(watches.lock().unwrap().is_empty());
+        changed(&watches, "/mailbox/d");
+        assert!(
+            tokio::time::timeout(std::time::Duration::ZERO, n.notified())
+                .await
+                .is_err()
+        );
+    }
+    #[test]
+    fn self_alias_and_normalization() {
+        assert_eq!(path("writer", "/self/../self/a/"), "/proc/writer/a");
+        assert_eq!(path("writer", "/selfish/a"), "/selfish/a");
+        assert_eq!(path("writer", "/"), "/");
+    }
+}
+
+#[cfg(test)]
+mod lifetime_tests {
+    use super::*;
+    /// The real caller's ended event releases the registration, without a cancellation RPC.
+    #[tokio::test]
+    async fn caller_death_removes_watch() {
+        let temp = tempfile::tempdir().unwrap();
+        let runtime = crate::runtime::Runtime::new(&temp.path().join("cache")).unwrap();
+        let host = Host::test_world(&temp.path().join("world"), runtime)
+            .await
+            .unwrap();
+        let (tx, mut rx) = tokio::sync::mpsc::unbounded_channel();
+        host.register_native(
+            "holder",
+            Arc::new(move |ctx, _, _| {
+                let tx = tx.clone();
+                Box::pin(async move {
+                    tx.send(ctx.pid.clone()).unwrap();
+                    std::future::pending().await
+                })
+            }),
+        );
+        let h = host.clone();
+        let task = tokio::spawn(async move { h.call("holder.hold", json!({})).await });
+        let pid = rx.recv().await.unwrap();
+        let mut watch = Box::pin(host.watch(&pid, Some(json!("watch")), "/mailbox"));
+        assert!(
+            watch
+                .as_mut()
+                .poll(&mut std::task::Context::from_waker(std::task::Waker::noop()))
+                .is_pending()
+        );
+        assert_eq!(host.0.watches.lock().unwrap().len(), 1);
+        host.call("core.kill", json!({"pid":pid})).await.unwrap();
+        watch.await.unwrap();
+        assert!(host.0.watches.lock().unwrap().is_empty());
+        assert!(task.await.unwrap().is_err());
+        host.stop().await;
+    }
+}

cli/tests/host/activation.rs

@@ -160,9 +160,9 @@ impl Drop for TestWorld {
         let _ = std::fs::remove_dir_all(&self.0);
     }
 }
-/// Builds a method-only document without an upgrade delay.
+/// Builds a method-only document; all plugins use the host's five-second grace.
 fn document(name: &str) -> Value {
-    json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}],"x-sausage-grace":0})
+    json!({"methods":[{"name":name,"params":[],"result":{"schema":{"type":"string"}}}]})
 }
 /// Adds an import of a provider's method list.
 fn consumer(provider: &str, expected: Value) -> Value {
@@ -484,7 +484,7 @@ async fn review_dropped_activation_completes() {
     host.stop().await;
 }
 /// PROTOCOL 10.1: discovery fails after its fixed ten-second deadline and receives EOF.
-/// PROTOCOL 3.7: old processes retain their declared grace while new calls use new code.
+/// PROTOCOL 3.7: old processes get five seconds regardless of declarations; new calls use new code.
 /// PROTOCOL 3.7: retirement waits for all live admissions, including reverted versions.
 /// PROTOCOL 3.7: rejected discovery candidates never receive retirement hooks.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -604,7 +604,7 @@ async fn source_changes_do_not_change_candidate() {
     );
     host.stop().await;
 }
-/// PROTOCOL 3.7: plugins without a declared grace period retain old calls before the default deadline.
+/// PROTOCOL 3.7: missing metadata also retains old calls until the uniform deadline.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn default_grace_retains_old_process() {
     let world = TestWorld::new();
@@ -623,6 +623,104 @@ async fn default_grace_retains_old_process() {
     assert_eq!(host.processes()[&pid].state, "running");
     host.stop().await;
 }
+/// PROTOCOL 3.7: metadata cannot shorten or extend the actual grace for active requests.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn uniform_grace_ignores_discovery_declarations() {
+    let world = TestWorld::new();
+    let declarations = [
+        ("missing", None),
+        ("default", Some(json!(60))),
+        ("long", Some(json!(3600))),
+        ("zero", Some(json!(0))),
+        ("negative", Some(json!(-1))),
+        ("invalid", Some(json!("long"))),
+        ("huge", Some(json!(1e300))),
+    ];
+    for (name, grace) in &declarations {
+        let mut old = document("work");
+        if let Some(grace) = grace {
+            old["x-sausage-grace"] = grace.clone();
+        }
+        world.plugin(name, old, document("work"));
+    }
+    let host = world.start().await;
+    let mut ready = host.subscribe();
+    let mut calls = Vec::new();
+    for (name, _) in &declarations {
+        let host = host.clone();
+        let method = format!("{name}.work");
+        calls.push(tokio::spawn(async move {
+            let result = host.call(&method, json!({})).await;
+            (result, std::time::Instant::now())
+        }));
+    }
+    // Guest notifications are events, not process/file polling.
+    for _ in &declarations {
+        let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
+            .await
+            .unwrap()
+            .unwrap();
+        assert_eq!(event["params"]["name"], "ready");
+    }
+    let before = std::time::Instant::now();
+    let plugins: Vec<_> = declarations
+        .iter()
+        .map(|(name, _)| json!({"plugin":name,"wasm":format!("/state/{name}/next.wasm")}))
+        .collect();
+    call(&host, "core.activate", json!({"plugins":plugins})).await;
+    // Use the existing discovery+stop test budget, not a new execution limit.
+    tokio::time::timeout(Duration::from_secs(15), async {
+        for task in calls {
+            let (result, ended) = task.await.unwrap();
+            assert_eq!(result.unwrap_err()["code"], -32002);
+            assert!(
+                ended.duration_since(before) >= Duration::from_secs(5),
+                "every declaration gets the full grace"
+            );
+        }
+    })
+    .await
+    .unwrap();
+    host.stop().await;
+}
+
+/// PROTOCOL 10.4/3.7: a watcher refresh uses the same policy as explicit activation.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn manual_copy_uses_uniform_grace() {
+    let world = TestWorld::new();
+    let mut old = document("work");
+    old["x-sausage-grace"] = json!(3600);
+    world.plugin("notes", old, document("work"));
+    let host = world.start().await;
+    // Wait for initial discovery without changing the installed version.
+    call(
+        &host,
+        "core.activate",
+        json!({"plugin":"notes","wasm":"/plugins/notes/plugin.wasm"}),
+    )
+    .await;
+    let mut ready = host.subscribe();
+    let work_host = host.clone();
+    let work = tokio::spawn(async move { work_host.call("notes.work", json!({})).await });
+    let event = tokio::time::timeout(Duration::from_secs(30), ready.recv())
+        .await
+        .unwrap()
+        .unwrap();
+    assert_eq!(event["params"]["name"], "ready");
+    let before = std::time::Instant::now();
+    std::fs::copy(fixture(2), world.0.join("plugins/notes/plugin.wasm")).unwrap();
+    assert_eq!(
+        tokio::time::timeout(Duration::from_secs(15), work)
+            .await
+            .unwrap()
+            .unwrap()
+            .unwrap_err()["code"],
+        -32002
+    );
+    assert!(before.elapsed() >= Duration::from_secs(5));
+    host.stop().await;
+}
+
 /// PROTOCOL 10.3: candidate imports must fit installed providers.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn checks_candidate_imports() {
@@ -818,11 +916,11 @@ async fn graph_missing_import() {
     assert!(dot.as_str().unwrap().contains("absent"));
     host.stop().await;
 }
-/// Checks the declared upgrade grace and dispatch to the new version.
+/// Checks that an explicit long declaration cannot extend grace; new calls use the new version.
 async fn old_process_grace_and_new_calls() {
     let world = TestWorld::new();
     let mut old = document("add");
-    old["x-sausage-grace"] = json!(0.5);
+    old["x-sausage-grace"] = json!(3600);
     world.plugin("notes", old, document("add"));
     let host = world.start().await;
     call(&host, "notes.hold", json!({})).await;
@@ -839,6 +937,76 @@ async fn old_process_grace_and_new_calls() {
         "2"
     );
     marker(world.0.join("state/notes/ended-1"), "ended").await;
+    let notice: Value = serde_json::from_slice(
+        &std::fs::read(world.0.join(format!("state/notes/superseded-{pid}"))).unwrap(),
+    )
+    .unwrap();
+    assert_eq!(notice["params"]["current"], fixture_hash(2));
+    host.stop().await;
+}
+/// PROTOCOL 3.7: every old process gets the new hash while still running, before grace ends.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn superseded_notifies_each_old_process() {
+    let world = TestWorld::new();
+    let mut old = document("hold");
+    old["x-sausage-grace"] = json!(60);
+    world.plugin("notes", old, document("hold"));
+    let host = world.start().await;
+    call(&host, "notes.hold", json!({})).await;
+    call(&host, "notes.hold", json!({})).await;
+    let pids: Vec<_> = host
+        .processes()
+        .into_iter()
+        .filter(|(_, process)| process.method == "hold")
+        .map(|(pid, _)| pid)
+        .collect();
+    assert_eq!(pids.len(), 2);
+    activate(&host, "notes").await.unwrap();
+    let expected =
+        json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":fixture_hash(2)}})
+            .to_string();
+    for pid in pids {
+        // The notice arrives while the old process is still in its five-second window.
+        marker(
+            world.0.join(format!("state/notes/superseded-{pid}")),
+            &expected,
+        )
+        .await;
+        assert_eq!(host.processes()[&pid].state, "running");
+    }
+    assert_eq!(
+        call(&host, "notes.version", json!({})).await["version"],
+        "2"
+    );
+    host.stop().await;
+}
+/// PROTOCOL 3.7: a service that exits on the notice restarts before its grace deadline.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn superseded_service_exits_early() {
+    let world = TestWorld::new();
+    let mut old = document("serve");
+    old["x-sausage-grace"] = json!(60);
+    world.plugin("notes", old, document("serve"));
+    std::fs::write(
+        world.0.join("config/boot.yaml"),
+        json!([{"method":"notes.serve","params":{"exit_on_superseded":true}}]).to_string(),
+    )
+    .unwrap();
+    let host = world.start().await;
+    marker(world.0.join("state/notes/service"), "1").await;
+    let pid = host
+        .processes()
+        .into_iter()
+        .find(|(_, p)| p.method == "serve")
+        .unwrap()
+        .0;
+    activate(&host, "notes").await.unwrap();
+    marker(world.0.join("state/notes/service"), "2").await;
+    assert!(!host.processes().contains_key(&pid));
+    assert!(
+        !world.0.join("state/notes/ended-1").exists(),
+        "old service must exit on the notice, not EOF"
+    );
     host.stop().await;
 }
 /// PROTOCOL 3.7: retirement receives the previous hash only after old processes end.
@@ -923,7 +1091,7 @@ async fn upgrades_enter_the_journal() {
                 .as_array()
                 .unwrap()
                 .iter()
-                .any(|e| e["line"] == "upgrade grace 60 seconds")
+                .any(|e| e["line"] == "upgrade grace 5 seconds")
             {
                 break entries;
             }
@@ -1047,6 +1215,36 @@ async fn old_consumer_can_call_removed_provider_method() {
     host.stop().await;
 }
 
+/// PROTOCOL 8: supervise actual WASI exits on either side of the readiness response.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn crashed_boot_service_restarts_before_and_after_ready() {
+    for when in ["before", "after"] {
+        let world = TestWorld::new();
+        world.plugin("notes", document("serve"), document("serve"));
+        std::fs::write(
+            world.0.join("config/boot.yaml"),
+            json!([{"method":"notes.serve","params":{"crash_once":when}}]).to_string(),
+        )
+        .unwrap();
+        let host = world.start().await;
+        marker(world.0.join("state/notes/service"), "1").await;
+        let entries = call(
+            &host,
+            "core.log",
+            json!({"grep":"service start notes.serve"}),
+        )
+        .await;
+        assert_eq!(entries.as_array().unwrap().len(), 2);
+        let processes = host.processes();
+        let services: Vec<_> = processes.values().filter(|p| p.method == "serve").collect();
+        assert_eq!(services.len(), 1);
+        assert_eq!(services[0].params, json!({"crash_once":when}));
+        let exits = call(&host, "core.log", json!({"plugin":"notes","grep":"exit 1"})).await;
+        assert_eq!(exits.as_array().unwrap().len(), 1);
+        host.stop().await;
+    }
+}
+
 /// PROTOCOL 10.2: activation and revert check interfaces but leave persistent state unchanged.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn state_is_outside_interface_checks() {

cli/tests/host/core.rs

@@ -208,7 +208,11 @@ async fn journal_recovers_after_sqlite_write_failure() {
     call(&h, "core.log", json!({})).await;
     let batches: Vec<Value> = std::fs::read_dir(w.0.join("logs/core/failed-batches"))
         .unwrap()
-        .map(|file| serde_json::from_slice(&std::fs::read(file.unwrap().path()).unwrap()).unwrap())
+        .map(Result::unwrap)
+        // The writer atomically publishes .json files; its temporary files can
+        // still be incomplete while later process-exit entries are spooling.
+        .filter(|file| file.path().extension().is_some_and(|ext| ext == "json"))
+        .map(|file| serde_json::from_slice(&std::fs::read(file.path()).unwrap()).unwrap())
         .collect();
     assert!(
         batches
@@ -1104,17 +1108,16 @@ async fn stop_closes_stdin_and_waits_for_exit() {
     let w = TestWorld::new();
     let h = w.start().await;
     let hc = h.clone();
+    let mut events = h.subscribe();
     let pending = tokio::spawn(async move { hc.call("probe.wait_eof", json!({})).await });
+    // Admission is not readiness: stop may otherwise close stdin before the
+    // request reaches the guest, so wait for the handler to enter its EOF loop.
     let pid = tokio::time::timeout(std::time::Duration::from_secs(20), async {
         loop {
-            if let Some((pid, _)) = h
-                .processes()
-                .into_iter()
-                .find(|(_, p)| p.method == "wait_eof")
-            {
-                break pid;
+            let event = events.recv().await.unwrap();
+            if event["params"]["name"] == "waiting-eof" {
+                break event["params"]["params"]["pid"].clone();
             }
-            tokio::task::yield_now().await;
         }
     })
     .await

cli/tests/host/main.rs

@@ -7,3 +7,4 @@ mod common;
 mod console;
 mod core;
 mod sdk;
+mod watches;

cli/tests/host/sdk.rs

@@ -178,6 +178,56 @@ async fn stop_runs_hook() {
     assert_eq!(std::fs::read(world.0.join("stopped")).unwrap(), b"yes");
     host.stop().await;
 }
+/// PROTOCOL 3.7: an idle SDK listener opts into drain and exits before EOF/grace.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn superseded_idle_listener_drains_before_grace() {
+    let world = TestWorld::new();
+    let host = world.start().await;
+    let (notice, mut notices) = tokio::sync::mpsc::unbounded_channel();
+    host.register_native(
+        "other",
+        std::sync::Arc::new(move |_, method, _| {
+            let notice = notice.clone();
+            Box::pin(async move {
+                assert_eq!(method, "superseded");
+                notice.send(std::time::Instant::now()).unwrap();
+                Ok(json!({}))
+            })
+        }),
+    );
+    let pid = call(&host, "probe.drain_service", json!({})).await;
+    // Replace with a different real component; force only skips unrelated test consumers.
+    std::fs::copy(
+        fixtures().join("notes.wasm"),
+        world.0.join("state/next.wasm"),
+    )
+    .unwrap();
+    call(
+        &host,
+        "core.activate",
+        json!({"plugin":"probe","wasm":"/state/next.wasm","force":true}),
+    )
+    .await;
+    // Activation preparation/discovery is outside upgrade grace. Time from the
+    // hook's outgoing call, not from the start of candidate compilation.
+    let before = tokio::time::timeout(Duration::from_secs(60), notices.recv())
+        .await
+        .unwrap()
+        .unwrap();
+    // Waiting on its lifetime lock is an exit event, not process-liveness polling.
+    call(&host, "core.lock", json!({"path":"/drain-owner"})).await;
+    assert!(before.elapsed() < Duration::from_secs(5));
+    assert!(!host.processes().contains_key(pid.as_str().unwrap()));
+    assert!(
+        !world
+            .0
+            .join(format!("stopped-{}", pid.as_str().unwrap()))
+            .exists(),
+        "drain must not run on_stop (discovery processes may run their own hooks)"
+    );
+    host.stop().await;
+}
+
 /// SDK framing carries payloads above 64 KiB in both directions.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn large_messages() {

cli/tests/host/watches.rs

@@ -0,0 +1,195 @@
+//! Real SDK watch callbacks and raw WASI filesystem mutations share the host namespace.
+use super::common::{CACHE, probe, sdk_probe};
+use sausage::{Host, world::World};
+use serde_json::{Value, json};
+use std::time::Duration;
+
+// Same call bound as core.rs; no sleeps or polling.
+async fn call(h: &Host, method: &str, p: Value) -> Value {
+    tokio::time::timeout(Duration::from_secs(20), h.call(method, p))
+        .await
+        .unwrap()
+        .unwrap()
+}
+async fn event(h: &Host, rx: &mut tokio::sync::broadcast::Receiver<Value>, name: &str) -> Value {
+    let result = tokio::time::timeout(Duration::from_secs(20), async {
+        loop {
+            let e = rx.recv().await.unwrap();
+            if e["params"]["name"] == name {
+                return e["params"]["params"].clone();
+            }
+        }
+    })
+    .await;
+    match result {
+        Ok(v) => v,
+        Err(e) => {
+            eprintln!(
+                "JOURNAL {}",
+                h.call("core.log", json!({"limit":100})).await.unwrap()
+            );
+            panic!("{e}");
+        }
+    }
+}
+async fn start() -> (tempfile::TempDir, Host) {
+    let dir = tempfile::tempdir().unwrap();
+    let world = World::create(&dir.path().join("world")).unwrap();
+    for (name, wasm) in [("probe", probe()), ("sdk", sdk_probe())] {
+        std::fs::create_dir_all(world.plugin_dir(name)).unwrap();
+        std::fs::copy(wasm, world.plugin_dir(name).join("plugin.wasm")).unwrap();
+    }
+    let host = Host::start_with_cache(&world.root, std::path::Path::new(CACHE))
+        .await
+        .unwrap();
+    (dir, host)
+}
+fn watch(h: &Host, path: &str) -> tokio::task::JoinHandle<sausage::RpcResult> {
+    let (h, path) = (h.clone(), path.to_owned());
+    tokio::spawn(async move { h.call("sdk.watch", json!({"path":path})).await })
+}
+async fn mutate(h: &Host, op: &str, path: &str) {
+    call(h, "probe.mutate", json!({"op":op,"path":path})).await;
+}
+/// PROTOCOL 4: pre-arm state is in the first scan; each post-scan mutation wakes
+/// the SDK without completing the watch. Covers descriptor/stream/truncate paths.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn armed_scan_and_later_changes() {
+    let (_dir, h) = start().await;
+    mutate(&h, "mkdir", "/state/mailbox").await;
+    call(
+        &h,
+        "probe.write",
+        json!({"path":"/state/mailbox/before", "text":"before"}),
+    )
+    .await;
+    let mut rx = h.subscribe();
+    let task = watch(&h, "/state/mailbox/../mailbox/");
+    let armed = event(&h, &mut rx, "armed").await;
+    assert_eq!(armed["entries"], json!(["before"]));
+    assert!(!task.is_finished());
+    for (op, expected) in [
+        ("descriptor", "descriptor"),
+        ("stream", "streamptor"),
+        ("append", "streamptorappend"),
+        ("size", ""),
+        ("descriptor", "descriptor"),
+        ("truncate", ""),
+    ] {
+        mutate(&h, op, "/state/mailbox/before").await;
+        let mut scan = event(&h, &mut rx, "changed").await;
+        while scan["text"] != expected {
+            scan = event(&h, &mut rx, "changed").await;
+        }
+    }
+    mutate(&h, "burst", "/state/mailbox").await;
+    let mut scanned = event(&h, &mut rx, "changed").await;
+    while !scanned["entries"].as_array().unwrap().contains(&json!("4")) {
+        scanned = event(&h, &mut rx, "changed").await;
+    }
+    while rx.try_recv().is_ok() {}
+    mutate(&h, "delete", "/state/mailbox/4").await;
+    let mut scanned = event(&h, &mut rx, "changed").await;
+    while scanned["entries"].as_array().unwrap().contains(&json!("4")) {
+        scanned = event(&h, &mut rx, "changed").await;
+    }
+    call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
+    assert!(task.await.unwrap().is_err());
+    h.stop().await;
+}
+/// PROTOCOL 4: atomic publication, rename source/destination, mounted paths,
+/// absent paths, and caller-death cleanup. No external host writes are required.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn rename_delete_and_mounts() {
+    let (_dir, h) = start().await;
+    for base in ["/state/inbox", "/run/inbox"] {
+        let mut rx = h.subscribe();
+        let task = watch(&h, base);
+        let armed = event(&h, &mut rx, "armed").await;
+        assert_eq!(armed["entries"], json!([]));
+        let staging = format!("{base}-staging");
+        mutate(&h, "mkdir", &staging).await;
+        call(
+            &h,
+            "probe.mutate",
+            json!({"op":"rename","path":staging,"to":base}),
+        )
+        .await;
+        event(&h, &mut rx, "changed").await;
+        mutate(&h, "atomic", &format!("{base}/published")).await;
+        let mut scan = event(&h, &mut rx, "changed").await;
+        while !scan["entries"]
+            .as_array()
+            .unwrap()
+            .contains(&json!("published"))
+        {
+            scan = event(&h, &mut rx, "changed").await;
+        }
+        assert_eq!(
+            call(
+                &h,
+                "probe.read",
+                json!({"path":format!("{base}/published")})
+            )
+            .await,
+            "complete"
+        );
+        while rx.try_recv().is_ok() {}
+        call(
+            &h,
+            "probe.mutate",
+            json!({"op":"rename","path":base,"to":format!("{base}-moved")}),
+        )
+        .await;
+        let mut scan = event(&h, &mut rx, "changed").await;
+        while scan["entries"] != json!([]) {
+            scan = event(&h, &mut rx, "changed").await;
+        }
+        call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
+        assert!(task.await.unwrap().is_err());
+    }
+    // /self watches bind to this caller, and mutations are visible via /proc/<pid>.
+    let mut rx = h.subscribe();
+    let task = watch(&h, "/self");
+    let armed = event(&h, &mut rx, "armed").await;
+    assert!(
+        armed["entries"]
+            .as_array()
+            .unwrap()
+            .contains(&json!("owned"))
+    );
+    event(&h, &mut rx, "changed").await;
+    assert_eq!(
+        call(
+            &h,
+            "probe.read",
+            json!({"path":format!("/proc/{}/owned", armed["pid"].as_str().unwrap())})
+        )
+        .await,
+        "visible via proc"
+    );
+    call(&h, "core.kill", json!({"pid":armed["pid"]})).await;
+    assert!(task.await.unwrap().is_err());
+    h.stop().await;
+}
+
+/// PROTOCOL 9: core discovery includes the armed watch and matches its proc projection.
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn discovery() {
+    let (_dir, h) = start().await;
+    let doc = call(&h, "core.rpc.discover", json!({})).await;
+    let projected = call(&h, "probe.read", json!({"path":"/proc/core/discover.json"})).await;
+    assert_eq!(
+        doc,
+        serde_json::from_str::<Value>(projected.as_str().unwrap()).unwrap()
+    );
+    let watch = doc["methods"]
+        .as_array()
+        .unwrap()
+        .iter()
+        .find(|m| m["name"] == "watch")
+        .unwrap();
+    assert_eq!(watch["params"][0]["name"], "path");
+    assert!(watch["description"].as_str().unwrap().contains("armed"));
+    h.stop().await;
+}

cli/tests/plugins/activation/src/main.rs

@@ -84,12 +84,52 @@ fn main() {
             .unwrap();
             json!({"retired":version})
         }
+        "work" => {
+            // Keep an actual request unanswered through grace; notices do not drain it.
+            send(
+                json!({"jsonrpc":"2.0","method":"core.notify","params":{"name":"ready","params":{}}}),
+            );
+            for _ in lines {}
+            return;
+        }
         "serve" | "hold" => {
+            // One genuine process failure, followed by a healthy service, without
+            // timers or liveness polling in either the fixture or supervisor.
+            let crash = request["params"]["crash_once"].as_str();
+            let first = crash.is_some()
+                && std::fs::OpenOptions::new()
+                    .write(true)
+                    .create_new(true)
+                    .open(format!("/state/{plugin}/crashed-once"))
+                    .is_ok();
+            if first && crash == Some("before") {
+                std::process::exit(1);
+            }
             send(json!({"jsonrpc":"2.0","id":request["id"],"result":{"version":version}}));
+            if first && crash == Some("after") {
+                std::process::exit(1);
+            }
             if request["method"] == "serve" {
                 std::fs::write(format!("/state/{plugin}/service"), version).unwrap();
             }
-            for _ in lines {}
+            for line in lines {
+                let Ok(message) = serde_json::from_str::<Value>(&line.unwrap()) else {
+                    continue;
+                };
+                if message["method"] == "rpc.superseded" {
+                    std::fs::write(
+                        format!(
+                            "/state/{plugin}/superseded-{}",
+                            std::env::var("SAUSAGE_PID").unwrap()
+                        ),
+                        message.to_string(),
+                    )
+                    .unwrap();
+                    if request["params"]["exit_on_superseded"] == true {
+                        return;
+                    }
+                }
+            }
             std::fs::write(format!("/state/{plugin}/ended-{version}"), "ended").unwrap();
             return;
         }

cli/tests/plugins/probe/src/main.rs

@@ -83,6 +83,9 @@ fn unknown_id() {
 }
 fn wait_eof(lines: &mut Lines) {
     std::fs::write("/state/waiting", "yes").unwrap();
+    send(json!({"jsonrpc":"2.0","method":"core.notify","params":{
+        "name":"waiting-eof", "params":{"pid":std::env::var("SAUSAGE_PID").unwrap()}
+    }}));
     for _ in lines {}
     std::fs::write("/state/stopped", "yes").unwrap();
 }
@@ -468,6 +471,7 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
             return None;
         }
         "network" => network(params),
+        "mutate" => mutate(params),
         "read" => read_file(params),
         "write" => write_file(params),
         "raw_call" => {
@@ -518,3 +522,68 @@ fn main() {
         }));
     }
 }
+
+/// Separate mutation calls let the host await a scan before each subsequent change.
+fn mutate(p: &Value) -> Value {
+    use wasip2::filesystem::{
+        preopens,
+        types::{DescriptorFlags as D, OpenFlags as O, PathFlags as P},
+    };
+    let path = p["path"].as_str().unwrap();
+    match p["op"].as_str().unwrap() {
+        "mkdir" => std::fs::create_dir_all(path).unwrap(),
+        "rename" => std::fs::rename(path, p["to"].as_str().unwrap()).unwrap(),
+        "delete" => std::fs::remove_file(path).unwrap(),
+        "rmdir" => std::fs::remove_dir(path).unwrap(),
+        "atomic" => {
+            let temp = format!("{path}.tmp");
+            std::fs::write(&temp, "complete").unwrap();
+            std::fs::rename(temp, path).unwrap();
+        }
+        "burst" => {
+            for i in 0..5 {
+                std::fs::write(format!("{path}/{i}"), "complete").unwrap();
+            }
+        }
+        op => {
+            let root = preopens::get_directories().remove(0).0;
+            let file = root
+                .open_at(
+                    P::empty(),
+                    path.trim_start_matches('/'),
+                    O::empty(),
+                    D::READ | D::WRITE,
+                )
+                .unwrap();
+            match op {
+                "descriptor" => {
+                    file.write(b"descriptor", 0).unwrap();
+                }
+                "stream" => {
+                    file.write_via_stream(0)
+                        .unwrap()
+                        .blocking_write_and_flush(b"stream")
+                        .unwrap();
+                }
+                "append" => {
+                    file.append_via_stream()
+                        .unwrap()
+                        .blocking_write_and_flush(b"append")
+                        .unwrap();
+                }
+                "size" => file.set_size(0).unwrap(),
+                "truncate" => {
+                    root.open_at(
+                        P::empty(),
+                        path.trim_start_matches('/'),
+                        O::TRUNCATE,
+                        D::WRITE,
+                    )
+                    .unwrap();
+                }
+                _ => panic!("unknown mutation"),
+            }
+        }
+    }
+    json!({})
+}

docs/SAU-384-rehearsal.md

@@ -0,0 +1,69 @@
+# Disposable handover rehearsal — SAU-384
+
+Authority: [/docs/design/plugin-handover/design.md](/docs/design/plugin-handover/design.md), Rollout; [/docs/symphony/artifacts/SAU-356/plan.md](/docs/symphony/artifacts/SAU-356/plan.md), Approval and operator gates.
+
+**Status: rehearsal complete, ready for review; not a live execution approval.** Rehearsal evidence and the reviewed target bundle are under [/docs/symphony/artifacts/SAU-384](/docs/symphony/artifacts/SAU-384). No shipped plugin or host behavior changes on this branch. The temporary migration implementation and native controller were executed only in a throwaway box and deleted after the successful migration verification. Their hashes/deletion record and source-as-log remain, not an executable migration tool.
+
+## Closed integration gaps
+
+- **Round 4, run 1392:** allocated-block/directory/sparse-copy space budgeting; disk-independent foreground progress/failure/pidfd-death monitor; incomplete pre-plan backup cleanup before abort writes plus low-space refusal; forward-only post-start continue with durable locked proof and stable continuation ID; guarded atomic-temporary cleanup; converter dry-run before freeze. Shared-filesystem ENOSPC (world/root/log at zero free bytes), each post-start RPC failure including lost committed send reply, real 120 s registration timeout, partial atomic crash/abort, wrapper-exec handshake/resume, sidecars and full workspace passed. Tools deleted after verified success. Source/logs, fixture failures and limits: [rework-1392.md](rework-1392.md). Live plan is still approval-gated.
+
+- **Round 3, run 1389:** deferred approved git 9cc9fa6f is bundled/preflighted, activated only after target cred is live and Surma unlocks. Real old/new Git + new cred loopback tests covered granted SSH, locked denial and new shared SSH. Added real-driver pre-start abort/restore/old-owner relaunch, live consistent-copy/executable-only caller guards, special-file/free-space gates, finite registration deadline/failure notification, exact pre-cut prev images and durable arm-marker crash recovery. Real partial-swap/ENOSPC/audit failures, arm interruption, final Herdr cutover/preservation/sidecars and full workspace (208 host tests) passed. Tool/controllers deleted after verification. Exact logs, selection and limits: [rework-1389.md](rework-1389.md); proposed live recovery/roots-side-effect plan: [runbook](runbook.md).
+
+- **Review rework, run 1381:** one detached external CLI now owns freeze→backup/plan/audit→apply→same-Herdr-pane launch and publishes a new explicit mediator verification message. Caller/controller death and interrupted planning resume passed. Plan/apply check the actual current-boot frozen checkpoint; old/target GC roots and permanent pinned owner wrappers, stopped-PTY TERM+CONT, SQLite sidecars and whole-world scale were tested. Tool/controllers were logged/deleted after verification; current workspace tests passed again. See [rework-1381.md](rework-1381.md) for exact evidence, limitations and disclosure of an early shared-Nix stale-root cleanup (all referenced targets/PIDs were absent on the host; subsequent tests used private Nix state). [Runbook](runbook.md) now assigns all downtime work to that external controller, not the cut old mediator. Live launch/release still require the explicit operator approval described there.
+- **Run 1373:** after mediator installed the observed public host key, approved read-only SSH inspection identified the actual Herdr pane, outer `sausage-loop`, world path, native architecture, old package/derivation/source and runtime closure. The actual old Nix package/loop → stop-all → offline swap → exact new Nix release package/loop passed in the disposable live-size world. A non-parent external controller separately proved loop-first stop, interruption/resumption, stale-PID refusal and safe frozen rerun. A real core.restart after loop exit could not respawn the host. See [native evidence](resume-1373.md).
+- **Run 1369:** armed/exit-race delay-gate tests passed on real core with actual target WASM; replacement orchestrator readiness/one-owner/all-watch arming passed. Hooks existed only in a disposable host copy, never the selected host artifact or committed code.
+- **Rollback:** actual old-manager/gateway tests prove **unsubscribe works; ID-only re-key does not**. The old binary ignores best_effort and replays history to a renamed ID. The runbook mandates the card-authorized unsubscribe option and prevents web re-registration during a session revert. A physical browser/SSE display is not claimed.
+- Final selected coherent bundle is **`cutover-targets-1389.tar.gz`** (SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`), **23 payloads**, including deferred approved **git 9cc9fa6f / 177 tests**. Same Nix native package/closure, approved cred main `0db40101` build **1636 / 274 tests**, other reviewed plugins/web assets and SDK `src/lib.rs`; no diagnostic debug host. Git is not swapped offline; Surma unlock precedes its mediator activation. Earlier bundles and runs remain historical evidence. Final exact-driver native/preservation/sidecar/failure and full-workspace gates passed again; executable/controller deletion and source-as-log are in rework-1389.md.
+
+The final frozen inventory/caller-schema/hash check remains an operator gate at the actual approved shutdown, not something a read-only rehearsal can perform in advance. Live execution still requires Surma's go and mediator/orchestrator action; no approval request is pending.
+
+## What was built and exercised
+
+- Host source `e09c4f9f0ef6b5833d6be066771eecd9e0a1f572`. Original debug native controller exercised real `sausage::Host`, VFS, watches, locks, service supervisor and WASM with synthetic/loopback effects. Run 1373 additionally built/rehearsed the actual **Nix release wrapper/package** `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; its empty-caller-PATH runtime check built/tested a real WASI+C plugin. Workspace suite passed again; all 105 Rust source files matched the tested tree. Neither host was released/deployed or selected by the live launcher. Native artifact/closure hashes are in `evidence/native-host-manifest.json`.
+- Reviewed session `cb844151` / SAU-409 final; orchestrator `c80a5854` / SAU-408; web `d883fbf2` / SAU-403; agent/board retained reviewed components. Fresh card-worktree builds: mail **1600 / 53 tests**, cred **1601 / 272**, cron **1602 / 24**, rooms **1603 / 59**, all passed. No activation. Component/source provenance and every bundled file hash are in `evidence/manifest.json`.
+- Frontend is exactly approved commit `3d76026749a5d21722033bbfe8743ac368946b0f`, retained production build from SAU-375 rework-1318. It remains `publish:hold`; no publication was attempted.
+- World SDK runtime sources already matched the selected host SDK. The only needed source-copy delta is **`src/lib.rs`**, documenting effective grace. Preserve world-owned Cargo manifests, relocated `interface`, and imports. A first broad SDK-copy attempt incorrectly included repository-relative Cargo paths; comparison caught this before any live change, and the final migration used only the one-file patch. Never copy the repository's SDK Cargo manifests over the world SDK.
+- Accepted SAU-362 byte snapshot: **1,491 sessions / 580,355,405 bytes**, including **70,689,796-byte default history**. Pristine snapshot remained unpreopened. Executable clones used synthetic model settings; their historical byte volume was retained. Source snapshot consistency is per-file, not a transactional live snapshot.
+
+## Successful checks and limits
+
+| Check | Evidence/result |
+|---|---|
+| Old ownership → stop-all → offline swap → target boot | Old `mail.start`, old session `send` and detached ledger processes inventoried. Native external controller awaited `Host.stop`, verified zero remaining processes and host exit, then backed up/transformed/swapped. Final shutdown **0.0958 s** in this fixture. One target manager, no mail deliverer, cred locked. |
+| Partial swap + rerun | Injected interruption after operation 3; hash-guarded completion and repeated installed-phase apply preserved all **4,374 non-target inventoried files**. No live/new work reset. |
+| Exact losses/preservation | Only `pending.jsonl`, `cancel`, and unclaimed pending non-Surma delivery rows selected for removal. Original mail messages, done/Surma copies and uncertain `unknown` deliveries retained without replay; scheduled SQL IDs, calendar/outcome fields and non-session JSON bytes retained. Synthetic grant DB bytes preserved. Loss inventory backs every removed file/row. |
+| Offline failure boundaries | Before-start rollback restored exact bytes; conflicting outside edits refused; ambiguous legacy `interrupt:true` refused without mutation. After-start apply and whole-world rollback both refused. |
+| Default/advisor HTTP replay | Three identical same-ID POSTs each, before/after consumption; exactly one answered run per session. Default returned null on repeats; advisor returned its same receipt. |
+| Explicit open target run | A run-start/consumed record with a finished-looking assistant tail, but no run-end, recovered under the **same agent-run UUID**, with one start. Legacy finished-looking history alone produced no target event/run. |
+| Real VFS/runner-boundary work | Actual SDK `armed` then plugin-write `changed`; five callback-boundary runs. Run 1369 additionally forced publication before runner `armed` and after its empty-mailbox exit decision while detached completion was held; no lost/duplicate inputs (disposable host delay gates, actual target WASM). |
+| Real core detached errors | Full nested error `data` survived a detached `core.call`. |
+| Real cron research deadline | **305.427 s** send→stopped in the final gate; real cron invoked warning and Stop. The 305-second test deadline is the existing 300-second warning interval plus design grace 5 seconds, not a new default. Held inference was cut, no follow-up run. Original configured research default remains 1,200 seconds. |
+| Effective grace / crash supervision | Passing native host suite exercises real WASI uniform-grace and before/after-readiness crash-supervision fixtures. Deliberate `core.kill` is NOT a crash: it disables that boot service; an initial rehearsal expecting its restart failed, correctly, and was replaced by the existing crash tests. |
+| Orchestrator upgrade/idle | Real file-watcher upgrade, FIFO singleton release, old exit before one replacement start event. Paused empty-board/live-size-world host CPU **0.02 s / 20.0002 s ≈ 0.10%**. This is whole-host idle CPU, not per-plugin CPU or live active-work load. Run 1369 additionally verified target acknowledgement, one owner and all 13 core watches armed after old exit. |
+
+### Live-size latency after real records exist
+
+After producing actual run-start/run-end/mailbox-consumed records, subsequent full boot measured **0.544 s** from manager service-start to the next boot service-start (includes discovery/ack overhead). Narrow registration catch-up samples ranged **0.0239 s to 12.0419 s** in the integrated world: the long sample coincided with other synthetic recovery/surfacing work, and the untagged sink cannot isolate attribution. Do not present this as a bounded manager-only benchmark.
+
+The clean final live-size end-to-end gate measured manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, and `mail.send` publication to default **0.5376 s**. The integrated post-record mail sample was **0.5235 s**. Both publication samples are below the existing orchestrator **3 s** and cred **10 s** caller budgets (SAU-400 report cites the originating SAU-366 review). These warm-cache synthetic-loopback real-core observations are not live latency guarantees. First scanning still delays first new input after the fast boot acknowledgement.
+
+## Resume verification
+
+Run 1369 forced recompilation and repeated the complete workspace suite on the restored original host source: pass, including **208 host integration tests**. All **144 source files** matched the selected source snapshot. Full log `/logs/box/2026-10-31/tc-asa3yr-exec-3838117964.stderr` (host tests lines 64–434; equivalence line 785). New tests and their limitations are detailed in `resume-1369.md`; new test controllers and instrumentation were deleted after successful checks. Original bundle and production source are unchanged.
+
+## Evidence and failures
+
+All following logs are under `/logs/box/2026-10-31/`:
+
+- Full workspace pass: `tc-asa3yr-exec-2320283969.stderr`. First cold-cache/full-load failure: `tc-asa3yr-exec-158345583.stderr`, missing offline async-task and a queued-build timing test. Populated the separate SDK-probe cache; entire suite then passed unchanged.
+- Final stop/swap/start: `tc-asa3yr-exec-1922130515.stderr`; offline fault/grant preservation: `tc-asa3yr-exec-616022371.stderr`.
+- Final live-size HTTP/deadline gate: `tc-asa3yr-exec-1243993190.stderr`. Earlier malformed mail `to` scalar, 30-second startup observation bound and blank JSONL parser mistakes were harness errors; corrected to a sequence, existing 120-second test budget and blank-line skipping. Final gate passed.
+- Open-run recovery and boundary runs: `tc-asa3yr-exec-903419814.stderr` (subsequent console-watch attempt failed). A console is not a running watch caller; the final SDK guest watch passed in `tc-asa3yr-exec-3261552495.stderr`. Same log retains the real upgrade trace and idle sample. An identical already-acknowledged registration correctly produced no second catch-up; the final probe used a fresh registration identity.
+- Actual old-manager unsubscribe/re-key comparison: `tc-asa3yr-exec-3307393394.stderr`.
+- Native artifact architecture/runtime paths: `tc-asa3yr-exec-899103518.stderr`; bundle build/CLI: `tc-asa3yr-exec-1075679130.stderr`.
+- Executed temporary tool and controller **as log only**, followed by deletion proof: `tc-asa3yr-exec-2902726782.stderr` (31,475 bytes). `evidence/tool-deletion.json` lists removed executables. No transition code was put in a shipped plugin.
+
+The initial broad-copy fixture and earlier failed tests are not the final kit. `evidence/migration-plan.json` identifies the final verified migration. Provider billing/cancellation, power loss, hung host filesystem behavior, physical browsers and independent per-plugin recovery follow-ups are not claimed complete. No live DB was opened; the accepted session snapshot was reused. No live config/SDK change, stop, activation, release, restart, push, deployment or `/www` write occurred.
+
+See [operator runbook](runbook.md) for exact owner/package/selector steps, frozen-audit and approval gates. [Run 1373](resume-1373.md) records the final native rehearsal and remaining unverified live-only boundaries.

docs/SAU-384-resume-1369.md

@@ -0,0 +1,39 @@
+# SAU-384 resume — run 1369
+
+Authority remains the card, design Rollout and SAU-356 approval gates. No live cut-over has been authorized or executed.
+
+## Results added
+
+- **Armed race passed:** actual target session WASM on real core, held immediately after core registered the runner's watch but before `armed`; a second real `session.send` published during the hold. Both IDs were consumed once in one run.
+- **Exit race passed:** held only delivery of the runner's initial result after the target's empty-mailbox exit decision. A second real publication arrived while the manager still considered that runner busy. Releasing detached completion produced exactly two runs, each consuming its own one ID. At publication the original runner had already exited; only the manager remained. No new timer/polling was added.
+- **Orchestrator upgrade readiness passed:** actual old SAU-370 component → selected SAU-408 component through the native plugin file watcher, old exit at 13:23:42.625Z before target start at 13:23:42.747Z, successful target acknowledgement and exactly one new owner. All 13 paths in `service.rs::WATCHES` armed in the target process. Prior whole-host idle duty measurement remains ~0.10% over 20 s in the paused, empty-board/live-size fixture; not a live workload guarantee.
+- **Actual gateway rollback passed:** new gateway first made its real `web-chat` best-effort registration; without another `web.start`, the session binary reverted to the actual pre-best-effort manager. Explicit unsubscribe caused zero `web.session_event` calls through the old manager's catch-up barrier. ID-only re-key caused one historical gateway callback. Therefore use the card's **unsubscribe** option; do not offer ID-only re-key as safe. This checks the actual callback/endpoint, not a physical browser/SSE display.
+- Restored production host source after disposable observation/delay hooks, forced recompilation, and reran `cargo test --workspace`: pass, including 208 native host integration tests and grace/supervision tests. All 144 source files matched the selected source snapshot. No source behavior changed in any committed plugin or host.
+
+## Evidence
+
+Under `/docs/symphony/artifacts/SAU-384/evidence/`:
+`resume-races-summary.json`, `resume-upgrade-summary.json`, `resume-upgrade-wire.json`, `resume-rollback-summary.json`, `resume-tool-deletion.json`, `resume-source-equivalence.json`.
+
+Full logs under `/logs/box/2026-10-31/`:
+- `tc-asa3yr-exec-1457078935.stderr`: race passes, followed by the initial upgrade-observation failure. Immediate swap raced initial asynchronous graph discovery; no old-version discovery baseline existed yet. Next method waited once for discovery completion, not a liveness loop.
+- `tc-asa3yr-exec-870753431.stderr`: successful upgrade readiness/order.
+- `tc-asa3yr-exec-4006214778.stderr`: actual gateway unsubscribe/re-key comparison.
+- `tc-asa3yr-exec-621695466.stderr`: executed tests/controller and host-only instrumentation as log, then deletion. No executable one-off/controller copies remain.
+- `tc-asa3yr-exec-3838117964.stderr`: forced-recompile full workspace pass; host tests lines 64–434, source equivalence line 785. Earlier unforced repeat also passed but is not used to establish recompilation.
+
+The original migration tool remains deleted. Recreated only disposable test controllers to close the above gaps; deleted those too after verification. Retained target bundle hash/artifacts are unchanged. Source-as-log is evidence, not a retained executable migration kit.
+
+## Remaining work / approval wait
+
+Only the actual external launcher/native installation manifest is still unresolved. Source `/repos/nixenv/machines/citadel/default.nix:22–27` disables `services.sausage`; `/repos/nixenv/machines/nexus/service-sausage.nix:10–11` describes the current Herdr panel. These sources do not pin the live panel, world path, launcher command, restart owner, stop completion API, executable/runtime closure, or architecture. Do not substitute `systemctl stop sausage` or `core.restart`.
+
+Read-only systemd-web GETs confirmed no active system `sausage.service`, and user-manager journal entries mention `/home/surma/.config/herdr/herdr.sock`; they still do not identify the currently running panel. Sources used:
+- `http://citadel-systemd.nexus.hosts.10.0.0.2.nip.io:8081/api/units?type=service`
+- `http://citadel-systemd.nexus.hosts.10.0.0.2.nip.io:8081/api/logs?unit=user%401000.service&lines=100&boot=0`
+
+No attached grant existed. One credential request is pending: **055995a51608c57dd25b323bea731fcb**, `id_scout`, `sub:card:SAU-384`, card duration, for **read-only** Citadel launcher inspection. The initial attempt supplied the immutable item ID, but live request expects a name and rejected it before filing; retry with `id_scout` filed the sole request. Do not request again; reuse it on resume. A credential approval is not permission to stop/restart/deploy.
+
+After approval: inspect actual process/parent/Herdr configuration read-only; pin owner, world path, synchronous stop/start/completion mechanism and correct native package/closure. Finish/rehearse that specific outer host swap or report a precise unresolved interface. Do not silently claim that the existing target-host/native-controller rehearsal replaced the old native service package. The current local host bundle is an aarch64-linux debug-profile stripped binary, not a validated live installation package.
+
+The surviving throwaway box is `tc-asa3yr`; temporary build/state copies remain there until its TTL. It contains no executable migration controller. If it expires, restore from retained target bundle/evidence and read the source logs, not a shipped plugin compatibility path. No pushes, activations, restarts, releases, config/SDK mutation, or `/www` writes occurred in the live world.

docs/SAU-384-resume-1372.md

@@ -0,0 +1,32 @@
+# SAU-384 resume — run 1372
+
+No live operation is authorized or performed. The existing rehearsal and target bundle are unchanged.
+
+## Read-only launcher inspection: trust prerequisite
+
+The existing card-duration SSH grant was approved; no new credential request was filed. `ssh.exec` to `citadel.vpn.surma.technology`, login `surma`, with the read-only command `uname -m; ps -eo pid,ppid,user,args | grep -E '([h]erdr|[s]ausage)'` was refused **before authentication/command execution** because the host is unknown. The refusal reported:
+
+```text
+SHA256:Fj3jS6DuV9SAo4PVQl/hTLI9SZPqWDwEd7E5hemfFkE
+citadel.vpn.surma.technology ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB8gP37Pnb3EyyQnkR2VdeJNzGU3OAuyJc3E3a8XpDaV
+```
+
+The SSH skill permits installing this observed key without another person approval, but this card explicitly forbids worker live-configuration changes. Therefore the **mediator** must add the observed line to `/config/known_hosts`, preserving existing entries, then resume this worker. Surma does nothing. No private key was requested or fetched. `ssh.hosts_status` confirmed there is no already-trusted Citadel alias. The preceding run already tried the preferred read-only systemd-web route: it cannot identify the Herdr panel/native installation boundary. Do not substitute Nexus credentials, forward another host's SSH agent, bypass host checking, or treat this as permission to stop anything.
+
+After trust is installed: reuse the attached grant, inspect launcher/process parents/Herdr config read-only; pin actual world path, architecture, executable/runtime closure and event/completion stop/start interface; then build/rehearse that specific outer host swap in a disposable box. No live stop, restart, config change or deployment is part of the resume.
+
+## Caller-store inspection (not a frozen live audit)
+
+`evidence/resume-1372-caller-inventory.json` records a read-only observation via `cron.list`, `board.hook_list`, boot-file inspection and current-main schema inspection. All eight cron jobs currently have `next_run:null`; six are historical `session.send` jobs with the legacy inner `detach:true`. Convert their executable JSON fields once with the already-rehearsed cron procedure, preserving SQL IDs, schedule/outcome fields and prompt bytes. References to old RPCs *inside their prompt strings* are history/instructions, not nested executable configuration; do not rewrite or replay them.
+
+Five board hooks were observed. Only `mediator-done` calls `session.send`; its `{session,text}` payload already fits the target contract without legacy detach/notify/interrupt fields. Target board supplies a fresh per-attempt input ID and follow-up kind. No rewrite is needed for that observed hook. Other hooks target mail/orchestrator and must remain intact.
+
+The frozen operator audit must additionally include board `invariant_target.data` (an arbitrary RPC template without a read-only target-get API), hooks and historical `hook_attempts`, board subscriptions/batches, room workflow/action records, orchestrator run/skill-review/host-job records and target session event registrations if present. Distinguish resumable target inputs from old diagnostic snapshots and never infer safe replay from a stored call-looking string. Current schemas are pointers, not substitutes for predecessor decoding or the actual frozen bytes. This session does **not** claim that final audit complete.
+
+## Evidence/checks
+
+- SSH refusal/public trust metadata: `evidence/resume-1372-ssh-trust.json`. No secret/JWT is copied to artifacts.
+- Existing box `tc-asa3yr` was inspected, not restarted. Executed controllers remain absent, including Python cache contents. Inspection log: `/logs/box/2026-10-31/tc-asa3yr-exec-803562803.stderr`, lines 1–40; the initial directory-list command returned 2 only because optional guessed directories did not exist, next inspection succeeded.
+- Host worktree fetched origin and merged local `main`: already current at documentation commit `8bf2a3c661ead774689e508ed45f609b40f60dd1`. Initial merge used `target` instead of the typed `rev` argument and was rejected without changes; corrected call succeeded.
+- JSON/caller/trust/manifest assertions and controller/cache-absence checks passed: `/logs/box/2026-10-31/tc-asa3yr-exec-666665822.stderr`. All 20 retained target file sizes/SHA-256 matched the updated manifest: `/logs/box/2026-10-31/tc-asa3yr-exec-3598104138.stderr`. No executable test/controller was written for these inline checks.
+- No Rust/plugin behavior changed or rebuilt. Run 1369's forced-recompile workspace pass and builds 1600–1603 remain the relevant gates. Independent plugin recovery follow-ups remain independent.

docs/SAU-384-resume-1373.md

@@ -0,0 +1,74 @@
+# SAU-384 resume — run 1373
+
+**Ready for review, not live execution.** Authority remains the card/design Rollout and SAU-356 approval gates. No host/session/cred operation, push or web publication was authorized or performed. No shipped host/plugin behavior changed.
+
+## Actual owner and native package resolved
+
+After mediator installed the observed host key, reused the approved SSH grant for **read-only** inspection as `surma`. The actual owner is Herdr workspace **w7**, tab **w7:t5**, pane **w7:p6**. Its shell runs `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`. This is not a systemd Sausage unit. The loop uses a mutable remote flake, exits on zero and refreshes/restarts after any nonzero exit (its existing sleep is 2 s). Therefore stopping plugins or requesting core.restart is not a deliberate native stop boundary.
+
+Citadel is **aarch64-linux**. Observed old package, derivation, original source and wrapper/binary hashes are pinned in `evidence/resume-1373-inspection.json`. Read the old derivation's actual CLI source: SIGTERM awaits `Host.stop`; an already-requested restart can still return 75. Do not reuse observed PIDs as constants.
+
+Built the exact Nix **release** package from the unchanged card host source:
+
+- `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`
+- `bin/.sausage-wrapped` SHA-256 `683623c4c77dbcfff590c9f0aa78f045d07935a466836a099ccc53d7336853cb`
+- Package NAR and every closure path/hash: `evidence/native-host-manifest.json`. ELF is AArch64 with the pinned Nix glibc interpreter. The closure equals the observed old package's closure except for the host root itself.
+- Empty caller PATH runtime check **passed**, building/testing an actual WASI plugin with C and the packaged toolchain; it was rebuilt with visible output, not inferred from `--version`.
+
+The box shares Citadel's Nix store: this build made an inert store path visible there, **not** selected by the live launcher. No live package/configuration/boot/SDK was changed.
+
+## Native end-to-end and interruption tests
+
+The disposable live-size world reused the accepted SAU-362 snapshot (1,491 sessions / 580,355,405 bytes). Executed the actual old Nix package and inspected loop, actual old mail/session components and actual target gateway; then:
+
+1. Recorded old plugin ownership through the real gateway/proc projection.
+2. External controller terminated the **outer loop first**, awaited its pidfd exit, then SIGTERM'd the exact old host and awaited its exit. The host survived its parent until its own graceful shutdown; controller survived outside the group. One final group inventory was empty; no process polling.
+3. Backed up frozen state, applied the approved queue/mail losses and cron conversion, injected interruption after operation 3, completed and repeated the hash-guarded offline plan.
+4. Changed a static external selector from the old to the exact new package. Marked started before launch. Started the same actual loop with explicit `SAUSAGE_FLAKE=path:<selector>`.
+5. Verified the running OS executable was the new Nix binary; exactly one target manager, no old mail deliverer, locked cred, real subscriber-registration completion. After-start apply/whole-world rollback refused. Marked verified, then stopped cleanly.
+
+Separate **non-parent controller** test used the reusable one-off `native_freeze.py` positional interface, not Python's parent-only wait(): pidfds pin the old loop/host, boot ID/start ticks/executable/group guard identities, owner exit precedes host stop. It proved interruption after owner exit then resume, stale-PID refusal without stopping the host, successful freeze and harmless frozen rerun. Another real-core test requested **core.restart after loop exit**: host exited and no owner/process in the old group could restart it. These tests close the external-owner gap; earlier run 1369 remains the evidence for armed/exit races, all-watch orchestrator upgrade ordering, and actual gateway rollback.
+
+Native gate inventory contained 4,434 files. Of the 4,372 non-operation files, 4,370 stayed byte-identical after target activity; the only changes were the expected manager registration file and core journal. Histories/settings/mail/jobs/grants outside intended operations remained equal. Prior offline tests additionally proved exact losses, unknown mail retention, SQL ID/calendar/outcome preservation, grant preservation and rollback before start.
+
+## Caller-store observation
+
+Read-only SQLite connections (`mode=ro`, query_only, read transactions) inspected board, cron and rooms; JSON workflow/config stores were also inventoried without printing prompts or grants. See `evidence/resume-1373-inspection.json`:
+
+- Six legacy cron session templates need the already-tested one-time conversion. Do not execute historical jobs or rewrite RPC words inside their prompts.
+- Both board's mediator hook **and invariant target** are already target-valid `{session,text}` calls. Preserve their bytes.
+- Eight historical running board session-attempt snapshots are diagnostic/uncertain, **never replayed** (`hooks.rs` explicitly documents this). Zero board subscriptions/batches were observed.
+- One abandoned room has no pending turn/review/split. Orchestrator run/skill-review records have no embedded old session/agent call objects; zero host jobs and no live session-manager registrations were observed. Preserve typed workflow records.
+- Three board runs had no outcome at observation, including this worker. This is old in-flight work, not proof of settlement or permission to replay it.
+
+The operator must repeat this audit on the **frozen** backup at the approved shutdown. This changing live observation cannot be that future snapshot. Unknown executable references block preflight; no shipped compatibility path was added.
+
+## Selected review artifacts and live steps
+
+Selected coherent bundle: **`/docs/symphony/artifacts/SAU-384/cutover-targets.tar.gz`** (24,591,858 bytes), SHA-256 `1bbcae4ef615ca8394b0c53b9a408441a49f8a618f059335c3bc248d958977c4`. The 22 payloads are hash-pinned in `evidence/manifest.json`; bundle also carries that manifest. It contains the native NAR/closure manifest, reviewed plugins, approved web `3d760267`, and SDK **`src/lib.rs` only**. It excludes the earlier diagnostic debug host. Older `target-artifacts.tar.gz` remains immutable prior evidence, not the selected native install.
+
+[Runbook](runbook.md) now gives actual owner/world/package paths, exact loop-first stop/checkpoint interface, proposed external root `/home/surma/sau-384-cutover` (absent at inspection), pinned selector and restart command, five-entry boot order (session before web; no mail.start), frozen caller conversion, web-release/rollback boundaries and mandatory executable deletion. Surma's go precedes mediator's live core/session/cred operation; every push separately needs go. Only mediator/orchestrator lands/executes; SAU-375 stays publish:hold until session and web are active, then orchestrator.release.
+
+All five executed one-off/migration/test scripts were **deleted after verified success**; Python bytecode cache was empty. Hashes/deletion proof: `evidence/native-1373/native-tool-deletion.json`. Source is retained **as log only**, never as a shipped plugin or retained migration executable. The live mediator must reconstruct its reviewed temporary copy, execute only after go, and delete it plus executable caches after verification too. Static target selector configuration must remain for the pinned native loop; it is not a migration tool.
+
+## Exact checks/logs
+
+All full logs below are `/logs/box/2026-10-31/`:
+
+- Nix package build/CLI: `tc-asa3yr-exec-1149169379.stderr`; packaged empty-PATH WASI+C gate: `tc-asa3yr-exec-395020656.stderr`.
+- Actual native old stop/offline interruption/swap/new boot: `tc-asa3yr-exec-3989255528.stderr`.
+- Non-parent controller interruption/stale-PID/frozen-rerun: `tc-asa3yr-exec-4245550011.stderr`; real core.restart after owner death: `tc-asa3yr-exec-2683606349.stderr`.
+- Native architecture/closure equivalence/hashes: `tc-asa3yr-exec-1926991205.stderr`.
+- Repeated `cargo test --workspace`: **pass**, including 208 host integration tests; `tc-asa3yr-exec-844023236.stderr`. All 105 Rust sources matched package/workspace copies. Run 1369's forced-recompile/source equivalence remains the earlier independent gate.
+- Executed source-as-log and deletion: `tc-asa3yr-exec-2945036769.stderr`; 22 payload hashes/controller absence: `tc-asa3yr-exec-643070322.stderr`.
+- New inventories/plan/losses/target process records/closure/source equivalence: `evidence/native-1373/`.
+
+Prior builds remain mail **1600**, cred **1601**, cron **1602**, rooms **1603**; no plugin source changed. Their remote-less repos have no origin (fetch correctly rejected); merging shared local main was already-current for all four. Host fetch/merge was already current before work; final commit/fetch/merge is reported separately.
+
+Failed approaches were corrected, not blockers: one incorrectly transcribed grant was rejected before any SSH command; the exact attached grant succeeded. zsh wildcard expansion and missing PATH tools were replaced by explicit installed Nix paths. `core.ps` does not exist; the native gate switched to the actual gateway `/fs/proc` projection and passed. Nix store logs were unavailable, so rebuilt the runtime check with visible `-L` output. One evidence-copy Path/string expression failed; corrected and resumed without re-executing migration. Full failure logs remain under /logs.
+
+## Limits, not silently claimed complete
+
+No new production-latency guarantee is claimed. Prior post-record real-core live-size samples remain: final manager acknowledgement **0.6078 s**, registration catch-up **0.0128 s**, mail.send-to-default publication **0.5376 s** (integrated **0.5235 s**), below callers' existing **3 s / 10 s** budgets. The integrated catch-up outlier **12.0419 s** remains disclosed. Orchestrator idle whole-host CPU was ~**0.10%** in the paused fixture; prior real cron warning/stop took **305.427 s**, and detached error data passed. These measurements and their synthetic/warm-cache limits are in implementation.md.
+
+Live shutdown/frozen backup/push/activation/release/publication, physical browser display, power loss and provider cancellation/billing remain unverified/out of scope. Old unrecorded runs are not automatically inferred complete/resumed. Independent per-plugin recovery follow-ups remain independent; the design accepts first-update cuts until they land.

docs/SAU-384-rework-1381.md

@@ -0,0 +1,56 @@
+# SAU-384 review rework — run 1381
+
+**Ready for review, not live execution.** This changes only local documentation and disposable one-off tooling. No shipped host/plugin behavior, live boot/SDK/launcher, activation, restart, publication, release or push changed. Surma's go still precedes the mediator's live operation and every push.
+
+## Review findings addressed
+
+1. **No mediator needed during downtime.** Replaced the disconnected procedures with one external CLI, `driver.py start → arm → detached run → verify`. `start` acknowledges a new OS session and blocks it on an arm FIFO. After arm the detached driver owns native freeze, immutable/resumable full backup, planning/frozen audit, offline apply, selectors, `started`, launch and readiness. It uses the exact pinned Herdr pane/shell, then publishes a **new explicit default verification message**, with a stable UUID, because the cut old mediator run is not recoverable. The last real-Core/Herdr test killed the invoking shell shim, then killed the detached controller at frozen, resumed it, and reached target in the same shell. Actual target run-start and mailbox-consumed records named that new verification message once. A real live SSH launch was not performed; the runbook makes the two finite foreground SSH launch/arm commands, and the required explicit go for that exception to SSH's background-work policy, part of the proposed operator plan.
+2. **Actual frozen-state gates.** `Kit.plan`, apply, starting and pre-start rollback require the real `freeze.json`, phase frozen, current boot ID, matching world, dead pinned owners, empty old group, no subsequently running host, and unchanged checkpoint bytes. A caller-supplied stopped literal is gone. Captured/wrong-boot checkpoints, running owner, changed config/SQL projection and after-start resume refuse before mutation. Offline retries accept only their saved before/after images.
+3. **Persistent installation pins.** Driver registers/verifies GC roots for exact old/target/loop packages and creates old/target selector and owner artifacts before freeze, so pre-start recovery already has the explicit old owner. Both wrappers export their own pinned selector. Cold restart in the same Herdr shell, without ambient SAUSAGE_FLAKE, ran the exact target; the explicit old wrapper ran the exact old package on a separate old-backup clone, never over new target work. Runbook prohibits bare loop/remote-default starts and explains that core.restart restarts the pin, not self-updates. Future host updates need an approved package/root/selector change; roots are not removed with migration scripts.
+4. **Real PTY job control.** Used actual Herdr 0.9.0 and an interactive zsh job-control shell, not setsid+piped stdin alone. Loop was foreground process-group leader in the shell's session. Freeze sends TERM then CONT before pidfd wait. Separate injection left the real host in observed **T** state after outer-owner exit; TERM+CONT completed shutdown with no surviving group member. This exercises the stopped-state hazard; it does not claim every timing of naturally delivered SIGTTIN.
+5. **Planning, SQLite and scale.** Incomplete backup resumes the same inventory/backup without cleanup or replacement. Streaming hashlib.file_digest avoids whole-file memory reads; os.walk/copy preserve directories and non-followed symlinks (including broken links). Complete DB sidecar families are backed up physically, recovered only in disposable copies, staged with SQLite backup in DELETE mode, and old sidecars removed before main-image replacement. Committed WAL and hot uncommitted rollback-journal fixtures passed interruption/rerun, exact losses, preserved job IDs/calendar/outcomes/generic JSON/unknown mail, and exact pre-start rollback. A full logical-size/count/bucket/directory-matched **synthetic** replica measured the whole-world backup path below; it was not an exact live-byte snapshot.
+
+## Evidence and exact gates
+
+Evidence directory: `/docs/symphony/artifacts/SAU-384/evidence/rework-1381/`. It contains only JSON manifests/inventories/checkpoints/results, not migration executables. `sandbox-operator-manifest.json` records the tested invocation; **do not execute it live**. `live-manifest.template.json` deliberately requires fresh identities, typed caller review/guards and a new stable continuation UUID before approval. Frozen guard tests used the real cron table projection; the live table/file set must include the board/config/registration review described in the runbook.
+
+All logs below are `/logs/box/2026-10-31/`:
+
+| Gate | Full stderr log / result |
+|---|---|
+| Final caller-killed + controller-killed/resumed + interrupted planning + same-Herdr-pane target chain | `tc-asa3yr-exec-775822946.stderr` — pass. 4,436 frozen regular files / 627,157,113 bytes in this session-size integration fixture; backup/hash stage 4.617 s. Earlier final chains also passed (`…-2599916298`, `…-2393656781`, `…-2876439273`). |
+| Final process ownership/locked cred/new explicit verifier/preservation; stopped-PTY CONT; cold pinned target; explicit old package on old clone | `tc-asa3yr-exec-3942858286.stderr` — pass. `post-target-processes.json`, `post-preservation.json`, `post-continuation-records.json`, `post-pty-stop-continue.json`; only planned paths and enumerated new activity changed. |
+| WAL + hot journal + interruption + exact rollback, repeated | `tc-asa3yr-exec-1595763713.stderr` — pass; first pass `…-1628589419`. `db-sidecars.json`. |
+| Whole-world scale | `tc-asa3yr-exec-3620999999.stderr` — **138,444 files, 12,441,162,164 bytes, 45,310 directories**. Inventory **70.555 s**, copy plus double hash verification **135.385 s**, combined **205.940 s**. Exact functions used by Kit.plan; synthetic bytes/names, observed live size buckets/counts, warm-cache box. No cold-storage downtime SLA. |
+| Current source equivalence + cargo test --workspace | `tc-asa3yr-exec-430706202.stderr` — pass; 121 Rust/Cargo inputs plus root Cargo/flake inputs matched current card tree. **208 host integration tests**, lines **64–431**, plus all workspace suites/doc tests. No Rust host source changed. The final cred main refresh/rebuild and repeated gates are below. |
+| Negative config/frozen caller controls, selected bundle rehash, verify and deletion | `tc-asa3yr-exec-3977808892.stderr` — pass; `negative-controls.json`, `bundle-recheck.json`, `tool-deletion.json`, `executed-test-deletion.json`. |
+| Source-as-log reconstruction/hash/AST check; no scripts/bytecode recreated | `tc-asa3yr-exec-3884064136.stderr` — pass; `source-log-roundtrip.json`. |
+
+### Reconstructable, but deleted after success
+
+The executed tool was three small files, not a plugin or host patch. Exact source survives **as log only** in `tc-asa3yr-exec-3977808892.stderr`:
+
+- `cutover_once.py`: lines **7–152**, SHA-256 `202eb7e635d675bb3960317913ae154e4f699fb7f5fe25a53ef34c965d0869c5`;
+- `freeze.py`: lines **154–226**, SHA-256 `d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe`;
+- `driver.py`: lines **228–393**, SHA-256 `7a26cfba9873541404f9661d5b0117f566301d408448cbb509a965d7900037aa`.
+
+The verify CLI was actually executed, marked the rehearsal verified, logged/deleted these files, and removed caches. All executed test controllers were also logged/deleted; the named disposable Herdr server was stopped. Hash reconstruction was checked **in memory**, without recreating executable files. The live runbook mandates the same deletion. Permanent owner/selector artifacts are installation artifacts, represented as bytes/hashes in `permanent-launcher-artifacts.json`, not retained migration code.
+
+## Final main refresh and selected builds
+
+The final required merge found cred main advanced to approved **SAU-418, `0db401012cae99b4424670d5c07a346281e30e75`** (plain anyone-may-use switch/shared SSH metadata and locked no-candidate fetch behavior). Inspected that diff and fast-forwarded the card worktree without discarding either side. No storage-schema or handover lifecycle delta was introduced. Rebuilt current cred: **1636 / 274 tests passed**, `/state/cargo/builds/1636/cred.wasm`; full output `/logs/cargo/2026-10-31/01a19281-a8ed-7127-914a-3ba4b5c15d17.{stdout,stderr}.log`.
+
+Final selection is now **`/docs/symphony/artifacts/SAU-384/cutover-targets-1381.tar.gz`**, **24,590,972 bytes**, SHA-256 **`9295c5fddbf5d68cbfd525602ec67e7dd895bf6afb99324191a6c514282eac57`**. Only the cred payload/provenance changed; all other 21 payloads, native host and SDK selection remain as reviewed. The original bundle remains immutable prior evidence. Current `evidence/manifest.json` and `evidence/rework-1381/main-refresh/selection.json` pin the final bundle. Host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`; SDK-copy **src/lib.rs only**. Mail/cron/rooms remain **1600/1602/1603**; approved web **3d760267** stays held until mediator/orchestrator release. No plugin implementation edits by this worker.
+
+Reconstructed the same three tool sources from their hash-checked log solely for this required new-main rerun, then deleted them again after verified success. Repeated caller/controller death/resume and actual Herdr cutover with the new cred bundle passed (`tc-asa3yr-exec-629237259.stderr`, same log directory). The first preservation assertion additionally encountered normal core-journal WAL/SHM activity; explicitly classified those journal sidecars as activity rather than migration loss. Final same-pane ownership/locked cred/fresh verifier/preservation/stopped-PTY/cold target/old-clone gates, negative controls and full `cargo test --workspace` all passed **`tc-asa3yr-exec-4223877022.stderr`** (208 host integration tests). Sidecar/rollback tests also repeated successfully. Final verify/source-as-log/deletion proof: **`tc-asa3yr-exec-741401146.stderr`**, **`evidence/rework-1381/main-refresh/{tool-deletion,executed-test-deletion}.json`**. Tool source hashes are unchanged from the reconstruction list above; no executable/cache remains. Final live template and sandbox manifests are under that `main-refresh/` evidence directory. Earlier replay/race/cron/watch/scale gates remain cumulative evidence; they were not silently relabeled as freshly rerun.
+
+## Failures, corrections and limits
+
+- Whole-tree transfer through box.put_tree rejected the 64 MiB service cap for repos/plugins/logs. Next method: read-only host size histogram, then regular-file size-bucket replica and the exact backup routines. This measures whole-world **scale**, not original live contents/path topology. The original accepted session byte snapshot still supplies the real-history integration volume.
+- Initial PTY fixtures lacked state/core for console history, then matched an http-prefixed address against a bare address. Corrected fixture and barrier. Typeahead from two early console commands lost the second; final driver queues **one** subscription command. An early Herdr multiline output matcher stalled; inspection showed Herdr wait-output itself polls, so final code removed it entirely in favor of inotify completion. The actual registration path is `state/session/manager/registrations/subscriptions.json`, now corrected.
+- **Shared-store foot gun:** an early default-store root query pruned **1,144 stale auto-root links and 571 stale temporary-root files** in shared Nix state. This was not an authorized live-config operation and no GC/package deletion was requested. Immediately stopped that method. Read-only SSH inspection on Citadel found **none** of those referenced targets and **none** of those PID numbers existing on the host, so no active roots required restoration. All subsequent root tests used a **private local Nix state database**, loaded from read-only closure metadata; no shared root query was repeated. Exact evidence/log: `gc-root-incident.json`, `tc-asa3yr-exec-2216542428.stderr`. This incidental stale-registration cleanup is disclosed rather than claiming literally zero host-state writes.
+- One post-start assertion read history before the new verification message was consumed. Replaced it with arm-then-scan/history-write completion. In the final gate publication→run-start was **23.753 s**, →consumption **24.530 s**, queued behind startup reconciliation. This is **not** mail.send publication latency. Initial acknowledgement is not full scan completion.
+
+Prior accepted replay/race/rollback/deadline/grace/crash-supervision/event-driven orchestrator evidence remains unchanged. Post-record manager ack **0.6078 s**, catch-up **0.0128 s**, mail publication **0.5376 s** / integrated **0.5235 s** remain below existing **3 s / 10 s** publication budgets; integrated catch-up outlier **12.0419 s**, ~0.10% idle duty, detached error propagation and **305.427 s** real cron warning/stop remain disclosed in implementation.md. This rework does not replace those measurements with a claim that the initial scan is fast.
+
+Still unverified/out of scope: actual approved live cut-over/frozen backup and complete production caller guards, live SSH detach/Herdr w7:p6 execution, orchestrator.release/publication/physical browser, cold-storage timing, power loss, provider cancellation/billing and independent recovery follow-ups. The live launch exception needs explicit Surma go as part of the proposed plan; no such approval is claimed or requested by executing anything here.

docs/SAU-384-rework-1389.md

@@ -0,0 +1,53 @@
+# SAU-384 round 3 — run 1389
+
+Ready for review; **not authorization for live execution**. Addresses `/docs/symphony/artifacts/SAU-384/extra-round-3.md` and review 1388. Only documentation and disposable tooling changed; no transition code in any shipped plugin/host.
+
+## Reviewed selection and proposed plan
+
+- Selected bundle: `cutover-targets-1389.tar.gz`, **27,271,968 bytes**, SHA-256 **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**. Embedded manifest SHA-256 **b48f2dacda3f348a67574767c6accfae1f1102227f94f86868ec1436bd4df394**; all **23** payload sizes/hashes checked. Prior bundles are immutable historical evidence.
+- New deferred payload: approved git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests, not rebuilt here). Staged world copy: `/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm`. Both manifest `deferred_plugins` and operator `deferred_artifacts` are hash-checked before freeze. **Not installed offline**: mediator activates it only after target cred is live and Surma unlocks through the UI. The runbook uses the real `core.activate` **wasm** argument and includes the activation check before web release/final deletion.
+- Cred remains approved **0db40101**, build **1636 / 274 tests**. Mail/cron/rooms remain **1600/1602/1603**. No new plugin build/source edit this run. Native host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, source e09c4f9f; old host mggvibk8 and loop pz2b94a2 remain pinned. SDK-copy list: **src/lib.rs only**. Web remains approved **3d760267**, release held until session/web/verification prerequisites.
+- Exact external commands, operator/approval boundaries, checkpoint archive/retry and rollback rules: [runbook.md](runbook.md). Live manifest template: `evidence/round3-1389/live-manifest.template.json`. Resolve fresh identities/caller guards before go; sandbox manifests are evidence, not live instructions.
+- **Expected live Nix side effect requiring inclusion in Surma's go:** three `nix-store --query --roots` calls, for old host, target host and loop, may prune stale auto-root and temporary-root registrations. They do not delete store packages. Five root registrations (those three plus controller/Herdr) remain in the proposed operation. This round used only the **private rehearsal Nix DB**; no shared/live roots query was repeated. Prior incident remains disclosed in rework-1381.md.
+
+## Six findings fixed
+
+1. **Git and interim compatibility.** Actual live old WASM (reported source f089b56d; SHA-256 4b460bce9b67552e223bb209591009056d409b77290182f6cf142adc8c4c97f4) ran against real new cred in a separate synthetic world. Explicit-grant SSH clone authenticated with a real loopback sshd, recorded exactly one sign use and fetched the synthetic commit. Locked use returned `vault_locked`. Old grant-free shared SSH still fails requesting a local key; old grant-free HTTPS retains its old config/anonymous behavior, not the new sharing switch (source checked). Old granted HTTPS reached/was accepted by new cred and reserved one use, then the deliberately closed loopback destination returned `network_error` plus capture; **successful HTTPS transfer is not claimed**. After actual synthetic unlock and new-git activation, shared grant-free SSH clone succeeded. Do not schedule Git network work in the locked/old-git interval, introduce key/config-secret workarounds, or blindly retry an already-started caller's errors. Local status/commit/main merges are unaffected. Live secret-backed operations/pushes remain untested and separately approval-gated.
+2. **Pre-start abort and human operator.** `check`/start, arm and immediate pre-freeze gates inspect consistent live SQLite backup-API copies, special file types, caller/file/artifact guards and free space. Space budget derives from current regular-file logical bytes plus the actual staged plugin/prev/SDK and recovery/stage/audit DB copies; no arbitrary multiplier/headroom/default. Growth or later ENOSPC is still possible. Driver now has a real `abort` CLI: exclusively lock, finish freeze if begun, restore only approved before/after paths from the complete immutable backup, verify full frozen inventory, then checkpoint-before-launch the explicit old owner in the same pane. No completed plan means apply never began: do not restore an incomplete backup. Restore is resumable; after `old_start_attempted` inspect instead of launching twice. Surma is the named external operator, with independent terminal log monitoring, `failure.json`, and best-effort Herdr notification (submission result retained; physical display not claimed). Runbook gives exact resume/abort and evidence-archive/re-resolution commands. After target `started`, automatic resume/restore/relaunch stays forbidden. Missing registration now exits on one absolute **120 s** deadline from cron/src/types.rs's existing registration/publication budget and the earlier real-core fixture bound; Herdr/RPC calls are finite too. No polling for quiescence or availability.
+3. **Executable-only SQL guards.** Cron projects id/method/params_json/cron/at/timezone/detach/call_timeout_secs, not next/last/result/occurrence bookkeeping. Board excludes last_sent/ended and batch IDs/lines/due; unions active subscription config with pending config snapshots so creating/delivering the same active configuration is not drift. New/changed executable configuration still refuses. Final automatic subscription removal changes the executable set and requires refresh or pre-start abort, explicitly documented. Add any planned mediator cron wake-up **before** resolving/approving operator.json. No caller/boot/plugin/SDK edits between resolution and freeze; arm checks again. No in-progress manifest edits.
+4. **prev contract.** Every offline-swapped plugin's prev is now its exact **frozen pre-cut current**, under before/after guards and restore. Tests checked all nine, including paths where an older prev existed. Git waits for normal later activation/prev behavior. This makes the fallback image meaningful, **not** a license for blind incompatible individual `core.revert`; coordinated rollback and unsubscribe-before-session-revert remain mandatory.
+5. **Roots side effect.** Listed above and in the live runbook/approval plan, rather than describing roots queries as mutation-free. Subsequent tests stayed private.
+6. **Arm race.** Replaced reader-dependent FIFO with atomic, manifest-bound **arm.json** and an arm-then-scan inotify watch. Idempotent arm returns with no reader. `resume` accepts waiting_for_arm; a kill after marker observation but before saving armed reuses the same marker. No manual FIFO writer/cleanup is needed.
+
+## Executed gates and full logs
+
+Prefix: `/logs/box/2026-10-31/tc-asa3yr-exec-`.
+
+| Gate | Log / outcome |
+|---|---|
+| Final exact tool, caller killed + controller killed/resumed, incomplete backup resume, real Herdr job-control stop/swap/start, same shell, post-start preservation/new verifier, prev hashes, pinned cold restart and old clone | **3699755631.stderr**, pass. Final backup 4,436 files / 627,157,113 bytes, 4.559 s (session-size integration, not full-world downtime). No old mail owner; one manager; cred locked. |
+| Live special-file/zero-space negative, mutable cron vs changed payload, pre-arm drift, frozen SQL mismatch -> real-driver abort/old relaunch, consumed-arm/save kill/resume, post-start abort refusal | **1485773773.stderr**, pass. Zero-space gate used a test-only mocked disk-usage response; actual ENOSPC tested below. |
+| Real-driver partial swap -> abort/restore/old relaunch; actual backup ENOSPC on a private tmpfs introduced after good preflight -> untouched-old relaunch; lost console subscription -> bounded clean exit | **4142682240.stderr**, pass. Registration wait 120 s; elapsed 132.493 s including backup/launch. Started checkpoint refused both resume and abort. |
+| Real old/new Git + new cred, loopback synthetic OpenSSH/HTTP negative, unlock then deferred activation | **99781693.stderr**, pass. No live key/grant/remote used. |
+| SQLite WAL + genuinely hot rollback journal, exact losses, preserved job IDs/calendar/outcomes/generic JSON/uncertain mail, apply interruption/rerun and exact physical restore | **1727732043.stderr**, pass. |
+| Board executable projections and old-select-star negative control; template/deferred-artifact checks; final driver's actual Herdr failure notification + abort | **1276236846.stderr**, pass. |
+| `cargo test --workspace` against unchanged production source | **2259546796.stderr**, pass: **208 host integration tests, lines 62–431**, plus all workspace suites/doc tests. 133 Rust/Cargo/flake inputs matched card tree. No Rust implementation change. |
+| Executed verify mode, complete source-as-log and deletion | **2013747101.stderr**, pass. See hashes/ranges below. |
+
+Prior accepted race/replay/deadline/core-watch/upgrade/idle/grace/crash-supervision evidence is cumulative, not silently rerun or claimed fixed here. Prior live-size scale sample remains **138,444 files / 12,441,162,164 bytes / 45,310 directories**, synthetic size-bucket match, 205.940 s inventory+copy/hash, warm cache. Post-record manager ack 0.6078 s, registration catch-up 0.0128 s, mail publication 0.5376 s (integrated 0.5235 s) remain under caller publication budgets **3 s orchestrator / 10 s cred**; catch-up outlier 12.0419 s and initial-verifier startup scan delay 23.753/24.530 s remain disclosed. No production downtime/latency guarantee.
+
+## Deleted executable and reconstruction receipt
+
+Only source-as-log remains in **2013747101.stderr**. Reconstruct exactly three files for the later approved operation, hash-check before use; do not reconstruct test controllers:
+
+- cutover_once.py: lines **2–164**, SHA-256 **3adf65ebb0b3401a0d1976524f35f7450e907ad32f907dcbb1cd2f7e0a7e5d3a**;
+- freeze.py: lines **166–238**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
+- driver.py: lines **240–503**, SHA-256 **686167318c84a32c1010daf4517f036bcb654384cadab987fa47fb13deecd2bf**.
+
+`verify` actually logged/deleted the executable after successful rehearsal. Executed test/patch/controllers and test launchers were logged/deleted too, with bytecode caches removed. `tool-deletion.json` and `executed-test-deletion.json` are under `evidence/round3-1389/`. Permanent pinned owner/selector configuration is not migration code; it stays rooted through the recovery window. The live runbook still requires mediator deletion after verified success.
+
+## Failures and limits
+
+Tried next methods, not blockers: initial generated SQL fixture had quoting syntax wrong (corrected); generated lockfiles made whole-directory diff reject (compared actual Rust/Cargo/flake input bytes, then ran full suite); final-sidecar extraction included a trailing log status (trimmed); a post-check initially read the previous round's stopped-world manifest (corrected to this round, then passed). Synthetic SSH setup needed explicit typed key metadata, a sandbox-only sshd account and an unlocked sandbox login account; default root was locked. A failed fixture left two loopback sshd listeners, later explicitly identified/stopped. Initial sandbox activation used `path`; core required `wasm`, corrected and the complete final Git gate passed. No failed test was relabeled successful; full errors remain in logs.
+
+Still unverified/out of scope: approved live cut-over/frozen caller snapshot, actual live SSH-detach/pane w7:p6 execution, Surma's real unlock/new-git live networking, physical Herdr notification/browser display, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. No live activation/restart/config/owner change, push, release, deploy or /www publication this run. Surma's go is still required before mediator core/session/cred operation, detached SSH exception, roots side effects and **every push**.

docs/SAU-384-rework-1392.md

@@ -0,0 +1,60 @@
+# SAU-384 round 4 — run 1392
+
+Ready for review; **not live execution approval**. Addresses all five findings in extra-round-4.md. No shipped Rust/plugin behavior changed. Current-main source and state contracts were rechecked; all work was in the card tree, review artifacts and disposable box tc-zg0j56.
+
+## Selection and operator plan
+
+The selected bundle is unchanged: `cutover-targets-1389.tar.gz`, **31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b**, 23 payloads. All payload sizes/hashes and the native closure hashes were freshly checked. Host remains `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0` from e09c4f9f. Cred remains 0db40101 / build 1636 (274 tests); mail/cron/rooms remain 1600/1602/1603. Deferred git remains approved 9cc9fa6f, activated only after target cred is live and Surma unlocks. No new WASM/package build. SDK-copy list: **src/lib.rs only**. Approved web 3d760267 remains held until the operator releases SAU-375 after session/web and verification prerequisites.
+
+Exact commands and approval boundaries are in [runbook.md](runbook.md). Current non-executable operator template: `evidence/round4-1392/live-manifest.template.json`. It still requires fresh caller/owner identities, guards and explicit Surma go. The only executable source is reconstructable **as log**, not retained as a tool. Surma is the external human recovery operator. The old mediator is deliberately cut; the driver publishes a new explicit default verification input, not inferred recovery.
+
+## Five findings fixed
+
+1. **Shared-filesystem ENOSPC.** Space preflight measures POSIX allocated blocks for files/directories/root/non-followed links, and charges dense filesystem-fragment-rounded size where copy2 would expand a sparse file. It includes the concrete staging/prev/SDK/DB-copy roles; receipts save all measurements. Removed the false never-underestimates claim: metadata/normalization/growth/competing writers remain uncertainty. A separate sparse/tiny/directory fixture copied **4,599,808 allocated bytes** from **12,288 source-allocated bytes**, below its measured **4,603,904-byte** budget. No live margin/default was invented.
+
+   Mandatory foreground `driver.py monitor` receives memory-only Unix datagrams for inventory/backup progress and failure, and kernel pidfd controller-exit completion. Every start/resume waits for its own monitor; arm requires it. Failure JSON/log writes are best effort and cannot prevent CUTOVER FAILED or the independent Herdr notification attempt. A terminal still reports controller death on SIGKILL; no heartbeat/polling loop. Notification submission was tested, but returned shown:false/disabled; physical display is not claimed.
+
+   Actual ENOSPC used **one shared tmpfs for world, external root and driver.log**, device 83, **0 free bytes**, capacity 254,504,960 bytes derived from measured fixture/budget allocations. It failed during a real backup write after good preflight. Monitor showed failure and death despite unwritable disk evidence. Pre-plan abort removes only the reproducible incomplete backup subtree before any checkpoint write, retaining planning/inventory evidence. If the competing writer still fills the disk, it refuses before saving/launching. After that known test file was freed, the same abort restored coherent old ownership and readiness. Complete backups are never deleted. Every abort (with or without a completed plan) now checks space before its first checkpoint save; a second shared-filesystem zero-free-byte fixture proved the complete backup/checkpoint remained exact, then old relaunch worked after freeing only the known competing test file. Space is checked again before old relaunch. This is not proof against every filesystem's ENOSPC/power-loss behavior.
+2. **Post-start forward completion.** New `continue operator.json` locks, checks the bound manifest, exact pane/shell, exactly one target host and target plugin hashes, and performs only missing/read-retry-safe post-start RPCs. It never launches/applies/restores. Actual locked cred.status is saved durably before unlock; subscribe uses the stable cutover registration; send reuses the exact continuation UUID/text. It reaches awaiting_verification, then normal verify/delete is reachable. Real failures of **cred.status, subscribe, and send**, including send committed before its acknowledgement was lost, all recovered with one run-start/consumption. A real **120 s** absolute registration timeout recovered the same way without another owner. The value remains cron/src/types.rs's existing registration/publication bound. Runbook gives exact continue command, RPC payload derivation, locked-proof/unlock ordering and local Surma fallback if live SSH signing is unavailable.
+3. **Atomic temporary cleanup.** A child was SIGKILLed inside atomic after writing a real partial plugin replacement temporary, before rename. Abort removes only a saved operation's newly created .cutover-tmp, absent in frozen inventory and prefix-matching an approved before/after image; unknown bytes/links refuse. Full frozen inventory equality passed inside rollback before old launch. Resume still overwrites/completes its own temporary. No arbitrary world cleanup.
+4. **Converter pre-freeze dry run.** check/start, arm and immediate pre-freeze checks use the same pure cron converter as plan, on SQLite backup-API memory copies. interrupt:true, invalid text and non-object params refuse without freeze/mutation. Negative control: round-3 checker accepted the same invalid object payload, new checker refused. No compatibility code in a plugin.
+5. **Roots/wake-up documentation.** Every controller resume repeats all three roots queries and their possible stale-registration pruning; this remains part of the proposed Surma-approved live plan. This round queried roots only against a private rehearsal Nix DB. No cron wake-up is needed: the stable explicit driver continuation owns handoff. A due-during-downtime job is skipped; a just-before-freeze input can enter an old pending queue that is intentionally dropped. Neither is replayed. The runbook disallows relying on guessed backup duration as readiness.
+
+Testing also exposed a real **Nix Python wrapper exec identity race** in parent-side process capture. The launcher now receives the child's actual post-wrapper identity over a pipe-completion handshake. Controller kill/resume proved exact identity, a fresh monitor requirement and successful same-pane continuation. No liveness/endpoint polling was added.
+
+## Full logs and evidence
+
+All log names below have prefix `/logs/box/2026-10-31/tc-zg0j56-exec-`. Evidence is `evidence/round4-1392/`.
+
+| Gate | Full stderr / outcome |
+|---|---|
+| Final shared world/root/log ENOSPC, pre-save incomplete-backup deletion, low-space refusal, old readiness; partial atomic SIGKILL/abort; all three post-start RPC failures; real registration timeout | **166388571.stderr**, clean combined pass on final tool source. Evidence under `final/`, including shared-enospc.json, atomic-abort.json, *-reviewed.json and registration-timeout.json. |
+| Complete-plan ENOSPC on one shared filesystem: pre-write refusal, checkpoint/full-backup equality, old readiness after known competing-data cleanup | **4094731865.stderr**, pass; `final/complete-plan-enospc.json`. |
+| Controller wrapper handshake, killed-controller pidfd, resume waits for fresh monitor, target completion | **537908850.stderr**, pass; `resume-handshake.json`, both monitor logs. |
+| Converter pre-freeze refusals | **1954198187.stderr**, pass; `dryrun.json`. |
+| Negative control + sparse allocation + all payload/native closure hashes + **133** final source/build inputs | **1314968883.stderr**, pass; `converter-negative-control.json`, `allocated-space.json`, `source-and-bundle.json`. |
+| Committed WAL/hot rollback journal; exact pending/cancel/mail losses; preserved mail/jobs/IDs/calendar/outcomes/non-target JSON; apply interruption/rerun and physical rollback | **2622539348.stderr**, pass; `sqlite-sidecars.json`. |
+| cargo test --workspace, unchanged final production source | **3315145279.stderr**, pass; **208 host integration tests**, result at line **433**, plus all workspace suites/doc tests. |
+| Fresh target one-manager/locked-cred/original-byte preservation/pre-cut-prev verification; actual verify CLI and complete tool/test deletion | **3030415406.stderr**, pass; `final/` final-ownership-preservation.json, verification.json, final-verified-driver.json, tool-deletion.json, test-deletion.json. Earlier verify/deletion pass: 1403220230.stderr. |
+| In-memory source-log SHA/AST reconstruction; executables/bytecode still absent | **2019074395.stderr**, pass; `final/source-log-roundtrip.json`. Earlier docs/JSON/absence checks: **2041020851.stderr**; final docs/JSON/absence/23-payload/133-source-input validation: **1011452644.stderr**, `final/final-validation.json`. |
+
+### Executed source deleted after verified success
+
+Reconstruct only these three files from **3030415406.stderr** for a later approved live operation; do not reconstruct tests:
+
+- cutover_once.py: source lines **2–189**, SHA-256 **84b13644bd80c6b331dcd8df9bf79339f62fedb99f7dd376535b2bca7b0afd96**;
+- freeze.py: source lines **192–263**, SHA-256 **d83d3d4035e5f59e219cff949be3455718a756a9c2ed4442119d4883a72fedfe**;
+- driver.py: source lines **266–669**, SHA-256 **5c3cba1a69b8b5e8f82ea83caa0697af494bf93dd0297f0c1b5231192cbb5fad**.
+
+After generalizing the no-write space gate to complete-plan and untouched abort too, the source was reconstructed for the final affected rerun, then the real verify command again marked that rehearsal verified, logged/deleted the three files and caches; all executed/reconstructed tests/patches/wrapper were logged/deleted too. Permanent owner/selector installation artifacts are distinct, not migration code. Live runbook still mandates deletion after verified success.
+
+## Failed approaches and limits
+
+- First interpreter alias linked only bin/, losing Nix Python's site-package context. Linked the complete existing interpreter environment; imports and gates then passed.
+- Cold workspace cache made six offline guest builds miss async-task; populated the disposable cache and reran the **whole workspace**, 208/208 host tests, unchanged source. Initial full log **2236066973.stderr**, missing dependency at lines 755–785; final pass above. No retry-without-diagnosis claim.
+- First shared filesystem test refused **before freeze** because actual source growth exceeded its provisioned measurement. Recorded that valid negative control, then provisioned the test from measured fixture/budget roles before inducing deliberate post-preflight pressure. A later test filled before the backup had copied any data: abort correctly needed a no-write space check before its first save; fixed and repeated with a before-backup event gate. Final shared test reached zero free bytes during copy and the final combined gate passed cleanly. Earlier fixture checkpoint-name collisions, an existing evidence-export directory and a busy mount affected cleanup/export only; used fresh identities/paths and explicit disposable-pane cleanup, never edited migration checkpoints. Mounts were left for disposable-box teardown, not live cleanup.
+- First post-start fixture omitted session prompt files, so it correctly recorded run-end error rather than mailbox consumption. Added synthetic prompts and repeated all three failure modes from fresh worlds; each then produced exactly one consumed continuation.
+
+This round's failure fixtures are **small synthetic real-core/Herdr worlds**, not another live-size benchmark. Prior accepted live-size race/replay/core-watch/upgrade/deadline/idle/grace/crash-supervision, scale and Git compatibility/unlocked activation evidence remain cumulative. Startup 0.6078 s, catch-up 0.0128 s, mail publication 0.5376 s vs callers' 3 s/10 s limits, catch-up outlier 12.0419 s and initial-scan delay remain disclosed in earlier reports; not remeasured or upgraded to guarantees here.
+
+Unverified/out of scope: actual approved live cut-over/frozen caller audit, live detached SSH/Herdr w7:p6 execution, live SSH signing through target cred, Surma's real unlock/new-git external networking, physical notifications/browser, release/publication, cold-storage timing, power loss, provider cancellation/billing and independent per-plugin recovery follow-ups. Surma's external terminal has exact continue/verify commands, so final completion is not dependent on the untested live signer. No push, live activation/restart/config/owner change, release, deployment or /www publication. Surma go still precedes mediator core/session/cred operation, the proposed detached-SSH exception/root side effects, and **every push**.

docs/SAU-384-runbook.md

@@ -0,0 +1,234 @@
+# One-off handover runbook — rehearsed; live approval required
+
+Authority: [design / Rollout](/docs/design/plugin-handover/design.md), [SAU-356 gates](/docs/symphony/artifacts/SAU-356/plan.md). Results: [initial](implementation.md), [races/rollback](resume-1369.md), [native package](resume-1373.md), [external driver rework](rework-1381.md), [round-3 recovery/git gates](rework-1389.md), [round-4 failure paths](rework-1392.md). This is a proposed live plan, **not authorization**. Workers never execute it. Mediator/orchestrator lands and executes only after Surma's go; every push separately needs a fresh go.
+
+## 0. Prepare everything while the old world is still available
+
+Do not begin downtime with scripts, caller decisions or launch instructions still to be assembled.
+
+### Reviewed selection
+
+- Bundle: `/docs/symphony/artifacts/SAU-384/cutover-targets-1389.tar.gz`, SHA-256 `31dbb95b796c1fedfd8c37a8b2354cb0977face8653345dc026cc1bbdefac18b`. It includes final-main cred `0db40101`, build **1636 / 274 tests**, preserving approved SAU-418 instead of downgrading it. Git **9cc9fa6f1c9f01ce1a53d8042623bde60e026637**, approved SAU-418 artifact `/state/orchestrator/builds/git/9cc9fa6f1c9f01ce1a53d8042623bde60e026637.wasm` (177 tests), is included as **deferred_plugins/deferred_artifacts**, not an offline replacement. It activates only after target cred is live AND Surma unlocks it in the UI. Earlier bundles are immutable historical evidence, not this selection. Verify all **23** payloads (including deferred git) against its embedded `manifest.json`. `target-artifacts.tar.gz` is older debug-host evidence, **not** this installation.
+- Native host: `/nix/store/spyd9rix6r38m6vxfppyy9z83yp5dh85-sausage-0.1.0`, aarch64-linux Nix release package from unchanged host source `e09c4f9f`. The embedded `native-host-manifest.json` pins the NAR, binary and complete runtime closure. Import the reviewed NAR if needed; never overwrite immutable store files or substitute a diagnostic host. Import/deployment is an operator action, never a worker action.
+- Old package for this observed baseline: `/nix/store/mggvibk8lcgv2qw28jimrl60mi0qm41p-sausage-0.1.0`. Re-identify it and the owner before go; observed PIDs are not constants.
+- Actual owner: Herdr **w7:p6**, tab w7:t5, not a systemd Sausage unit. Its loop is `/nix/store/pz2b94a2v99r3msljdz82h7n3i81hhh5-sausage-loop/bin/sausage-loop /home/surma/world/`. Read `herdr pane process-info --pane w7:p6` once to resolve current shell/loop/host identities. Do not close Herdr or unrelated panes.
+- SDK copy list is **`src/lib.rs` only**. Preserve world-owned Cargo manifests, relocated `interface`, imports and all other SDK bytes. Resolve a changed baseline before proceeding.
+- Web is approved commit `3d76026749a5d21722033bbfe8743ac368946b0f` (SAU-375), held by `publish:hold`. Do not release it before target session and web are active.
+
+### External driver and resolved operator manifest
+
+Use external root `/home/surma/sau-384-cutover`, outside the world and stopped process group. Stage the extracted bundle under `targets/`. Reconstruct **only** `cutover_once.py`, `freeze.py`, and `driver.py` under `tool/` from the source-as-log headings in `/logs/box/2026-10-31/tc-zg0j56-exec-3030415406.stderr`; check their hashes against `evidence/round4-1392/final/tool-deletion.json`. These are the executed round-4 versions (source lines 2–189, 192–263, 266–669 respectively), not earlier scripts. Hashes/ranges are in evidence/round4-1392/final/source-log-roundtrip.json. They have a real CLI entry point. Do not reconstruct rehearsal/test controllers.
+
+The tested interpreter is `/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3`, with **inotify-simple 2.0.1**. Resolve its closure availability before go. After go and before starting the driver, pin its package and the inspected Herdr package with host-visible GC roots for the operation (`nix-store --realise <exact-package> --add-root <external-root>/gc-controller --indirect`, and equivalently `gc-herdr`). These root writes/imports are included in the approved live plan, not preparation actions authorized to a worker. The driver itself registers and verifies **old host, target host and loop** roots at `<root>/gc-{old,target,loop}` before freeze. Use the **host namespace**, not a box talking to shared Nix state: `/rework` roots in a box are not host-visible. Do not run GC as a test. Keep old/target/loop roots and both selectors for recovery after tool deletion.
+
+Start from `evidence/round4-1392/live-manifest.template.json`, a **non-executable template**, and resolve every placeholder before approval:
+
+- native paths, Herdr binary/context, pane and current shell/loop/host PIDs;
+- external root, world, bundle, embedded manifest SHA, offline artifact paths **and deferred git**, SDK path, exact target boot list; root and world are on the same filesystem in this plan; resolve a different storage topology before go;
+- complete old in-flight `/proc/*/info.json` and native-owner inventory, plus uncertain external effects; a finished-looking assistant tail is not settlement;
+- a new continuation UUID, generated **once**, and the explicit new mediator verification instruction. Keep that UUID on all retries;
+- typed executable-caller review and its frozen guards below.
+
+Do not copy sandbox PIDs, `gc_store`, or any `test_*` gate into a live manifest. The template deliberately omits those test-only fields. Record/hash the fully resolved `operator.json`; driver checkpoints bind to its exact bytes. Resolve a schema/selection discrepancy while the old world is available, not during downtime.
+
+### Executable caller review and frozen guards
+
+Use predecessor formats: session source `f9122585` Paths/queue; cron `ba75c102` and current cron README one-off procedure. Do not substring-rewrite prompts or diagnostics. Review current stores as in `resume-1373.md`, then encode the **approved** file hashes and deterministic SQL projections in `caller_file_guards` and `caller_sql_guards`:
+
+- cron `state/cron/jobs.sqlite`: use the template's ordered `id, json_extract(data,...)` projection of **method, params_json, cron, at, timezone, detach, call_timeout_secs**. Do not hash next_run, last_run, last_result_json or last_occurrence_id;
+- board `state/board/board.sqlite`: ordered `id,data` for hooks/invariant_target; `id,target,session,data` for subscriptions; and `select id,data from subscriptions union select subscription,data from subscription_batches order by 1,2` for executable subscription snapshots. Ignore last_sent/ended, transient batch IDs/lines/due; a pending delivery of the same active configuration does not change this union. Historical hook-attempt snapshots remain diagnostics and **must not replay**. Automatic final subscription removal changes the executable set; resolve/refresh before arm, or take the pre-start abort path if it happens after arm;
+- boot and other discovered executable configuration/manager registrations. Review typed room/workflow/orchestrator stores too; preserve their histories/settings and non-executable records. Do not mistake prompt text for an RPC template.
+
+A SQL guard hashes `sha256(json.dumps(rows,separators=(',',':')).encode())`, where `rows = connection.execute(sql).fetchall()` in the approved ordered projection. JSON strings inside rows retain their exact bytes. These projections, including all column names, live in the current template and were tested against mutable cron/board bookkeeping. Any genuinely new executable store/field still needs review; do not silently omit it. The observed live baseline had **six cron templates to convert**, already-target-valid board hook/invariant calls, no subscription batches and no live manager registrations. This is an observation, not a future frozen snapshot. Unknown executable references, `session.work`, `session.cancel`, ambiguous `interrupt:true`, invalid text or unexplained notify destinations stop preflight for explicit resolution.
+
+**Before start, again before arm, and once immediately before freeze**, `check`/driver checks artifact and file guards, SQL projections on consistent live SQLite backup-API copies, special-file types (without hashing mutable bytes), space, and a **dry run of the same cron converter used by plan**. Object/text/interrupt refusals therefore occur before downtime, not just in plan. The dry run never rewrites the live DB. After freeze, guards run on disposable complete DB copies before apply; immutable backup evidence is never opened/recovered in SQLite.
+
+**Space is an allocated-block measurement, not a guarantee.** Root/world must have the same filesystem device in this reviewed layout. Walk regular files, directories (including the root) and non-followed symlinks; charge each at the larger of POSIX `st_blocks × 512` and its logical size rounded up to that filesystem's `f_frsize`. This includes sparse-file expansion by copy2, not just logical file bytes. Add the concrete staged plugin/prev/SDK allocations and the three existing DB-copy roles (recovery, staging, audit). Save source allocation, backup allocation budget, directory/file counts, fragment size and required/free bytes in `preflight-live.json`; refuse before freeze unless free covers that budget. The fresh sparse/tiny-file/directory fixture measured 12,288 source-allocated bytes and 4,599,808 copied bytes against a 4,603,904-byte budget (see evidence/round4-1392/allocated-space.json for actual measured values). **No claim that this can never underestimate:** metadata, DB normalization, growth, changed mounts and competing writers can still exhaust space. Resolve storage uncertainty in the approved maintenance plan, not with an invented margin.
+
+Add any genuinely new executable store to the manifest before arm. From resolution until freeze: no caller/boot/SDK/plugin changes; routine cron next/last/result and subscription delivery bookkeeping is permitted. If new intent is needed, cancel unarmed or abort pre-start, re-resolve, and approve the changed manifest; never edit a hash-bound in-progress manifest.
+
+**No cron wake-up is required:** the external driver publishes the explicit new mediator input. Do not rely on a timed wake-up for controller recovery. If a fallback wake-up is explicitly wanted, add/review it **before** resolving operator.json, not between resolution and freeze. It is useful only after the target is running and can accept new input. One due during downtime is skipped by cron; one firing immediately before freeze can enter an old pending queue that this approved cut deliberately drops. Neither is migrated/replayed. A guessed wall-clock time after the estimated backup duration is not readiness proof; use the driver's event/completion evidence and stable continuation instead.
+
+### Approval and external launch surface
+
+Surma's go must explicitly cover this resolved manifest, the stop-all core/session/cred cut, the detached external controller launch/arm, pinned launcher installation and the post-start verification/release/deletion plan. This card's outside-world-controller scope requires a controller that survives the mediator; it does **not** authorize executing it now.
+
+The proposed concrete execution surface is **two finite foreground `ssh.exec` calls as surma on citadel.vpn.surma.technology**, using the existing approved signing grant, with commands `cd /home/surma/sau-384-cutover && <the start command below>` and then the same prefix with `<the arm command below>`. Each command terminates; the reviewed native CLI, not a shell `&`, creates the detached OS child. **Ask Surma's go to explicitly authorize this planned exception to the ssh-exec skill's usual prohibition on creating background work through exec strings. Until that explicit authorization, do not launch.** Do not improvise a tunnel, SSH PTY, new worker session or an in-world controller. This is a fully prepared proposed live operation, not a claim that the current worker has permission or has tested a live SSH launch. The same finite foreground surface executes the final verify/delete command after target verification. After arm, loss of the old mediator/SSH RPC result is expected and does not terminate the already-detached controller; inspect the external checkpoint instead of retrying start.
+
+## 1. Surma monitors; mediator starts/arms the external controller
+
+Close ingress and record accepted first-update cuts before arm. **Surma is the external human recovery operator while the world is down.** He uses an independent Citadel terminal, never an in-world session/tool process. After mediator `start` acknowledges, Surma runs the foreground monitor below and leaves it visible. It binds a local Unix datagram socket and records the monitor/controller identities before arm. The controller waits for **its own** monitor on every start/resume; arm refuses without it. If an old monitor was killed, a new monitor removes its stale socket only after the saved exact process identity is dead; never unlink a live monitor's socket by hand.
+
+The monitor receives inventory begin/completion and completed backup-entry progress **without writing disk**, plus `CUTOVER FAILED` and the notification outcome. It also waits on the controller's **pidfd**: normal exit, SIGKILL or crash prints `CONTROLLER EXITED`, even when there is no log/failure.json. Progress datagrams are best effort; the independent death event is not. No heartbeat/liveness/endpoint polling or backup-duration timeout. Read `driver.json` and inspect the pane once after exit; distinguish awaiting_verification from failure instead of blindly restarting.
+
+Disk-backed `failure.json`, `failure-notification.json` and driver.log are best effort. Each may be unwritable at ENOSPC; none is a prerequisite for the failure signal or the Herdr notification attempt. One best-effort Herdr notification `Cut-over needs Surma` is submitted without first saving failure.json. Rehearsal submission returned shown:false/disabled; neither delivery nor a physical display is guaranteed. **Keep the external terminal monitor attached:** a silent tail of driver.log is not sufficient during backup. Whole-machine power loss can stop both controller and monitor and is not claimed covered.
+
+From the external root, on the approved host execution surface, mediator first runs the finite preflight command, then starts:
+
+```sh
+PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py check operator.json
+PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py start operator.json
+```
+
+`start` launches a new OS session with stdin closed and output in `<root>/driver.log`. A pipe completion handshake records the child's **actual post-Nix-wrapper-exec identity**, rather than racing an interpreter exec. It returns an acknowledgement and leaves the child waiting on inotify for its own monitor, then **durable arm.json**. There is no arm FIFO or reader-dependent write. Check that receipt/checkpoint once. Surma now runs in his independent terminal:
+
+```sh
+cd /home/surma/sau-384-cutover
+PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py monitor operator.json
+```
+
+Wait for `MONITOR attached; arm may now proceed` in that terminal. Then mediator runs the finite arm command:
+
+```sh
+PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py arm operator.json
+```
+
+`arm` requires the current monitor, repeats live preflight and atomically writes a manifest-bound marker. Re-arming is idempotent while waiting with that monitor; without it arm refuses finitely, never blocks on a missing FIFO reader. A crash after observing the marker but before saving `armed` resumes from `waiting_for_arm` and reads the same marker.
+
+From here **the external driver, not the old mediator**, owns steps 2–4. The old default run and its tool calls are deliberately cut. They are not recovered from legacy history. The driver uses an external advisory lock to exclude another controller. Rehearsal killed both the invoking caller shim and, separately, the detached controller after freeze; checkpoint resume completed the chain.
+
+## 2. Driver: freeze → full backup → plan/audit → offline apply
+
+The driver verifies reviewed bundle/plugin/SDK bytes and native closure hashes, registers real GC roots, and creates hash-guarded **old and target selector/owner artifacts before freeze**. It then:
+
+1. Pins boot ID, start ticks, executable and group for both native processes; confirms controller is outside their group. Signals the **outer loop first** and awaits its pidfd exit. Saves `owner_stopped`.
+2. Signals the exact host **SIGTERM then SIGCONT**, so a job stopped by background-PTY SIGTTIN can handle termination, and awaits host exit. Checks the saved group once for unexpected survivors; never blindly kills them. Saves `freeze.json` phase **`frozen`**, world path, boot ID and empty group inventory. No process/endpoint polling, sleeps or stop-request counting. Host shutdown stops all plugin/agent trees and suppresses restart admission. `core.restart` alone is not stop-all; after the loop is gone, exit 75 cannot respawn it.
+3. Makes the complete frozen backup outside the world. Streaming SHA-256 does not read whole files into memory. Directory inventory and copy preserve empty directories and symlinks **without following links**. An interrupted plan resumes the same immutable input inventory and partially completed backup; it neither overwrites an earlier backup nor requires deleting the plan directory. Progress is sent to the external terminal monitor. A pre-plan abort may discard only that reproducible incomplete backup, as described below. Unknown images refuse.
+4. Stages the approved transformations, then checks the frozen caller guards. **Both plan and apply independently require the actual frozen checkpoint**, same world/current boot, dead pinned owners, empty old group and no subsequently running host for this world. A caller-supplied `{remaining:0}` literal is not accepted. Changed checkpoint bytes also refuse.
+5. Stages **plugin.prev.wasm = that plugin's frozen pre-cut plugin.wasm** for every offline-swapped plugin, then target plugin.wasm. Both prev/current writes have before/after hashes and participate in restore. It never retains the unrelated older prev as the cut-over fallback. Git is deliberately untouched here; its later standard activation will save its own preceding binary. Applies hash-guarded before/after operations offline. File replacements fsync temporary bytes, rename, and fsync the parent. A third image refuses. If atomic replacement dies before rename, abort removes only that saved operation’s new `.cutover-tmp`, absent from the frozen inventory and byte-prefix-matching its approved before/after image; unknown temporaries refuse. Resume overwrites/completes its own temporary. `migration/plan.json` reaches `installed` only after all operations complete.
+
+Exact logical losses are **only** old session `pending.jsonl`, `cancel`, and unclaimed `mail.deliveries` rows with `state='pending'` and recipient not `surma`. Original messages, Surma copies, done rows and uncertain `unknown` deliveries stay; do not replay them. Loss records and original bytes/rows remain in backup/evidence. Histories/settings, scheduled SQL IDs/calendar/outcome fields, grants/requests and unrelated files remain intact.
+
+Cron conversion: require an object and nonempty text, refuse `interrupt:true`, remove false/absent legacy interrupt, inner detach/notify/permanent ID, set absent kind `follow-up`, outer detach false. Preserve other fields and non-session JSON bytes. Review removed notify destinations; no invented answer-forwarder.
+
+**SQLite sidecars:** copy the complete DB plus `-journal/-wal/-shm` family into immutable backup first. Recover only another disposable copy; use SQLite's backup API to stage a consistent DELETE-journal image. Plan removal of old sidecars **before** replacing the main DB, retaining the original physical family. This is normalization, not additional logical data loss. Tests covered committed WAL data, a genuinely hot uncommitted rollback journal, interruption and exact pre-start restore.
+
+Full-world backup is not instantaneous: the 138,444-file / 12,441,162,164-byte / 45,310-directory **size-bucket-matched synthetic** replica took **70.55 s inventory + 135.39 s copy/hash verification = 205.94 s** in the box. This includes repos/plugins/logs, unlike the earlier session-sized gate. It is warm-cache scale evidence, not an exact live-byte or cold-storage downtime guarantee. Allow that uncertainty in the approved maintenance plan; do not invent a timeout and kill the controller mid-copy. Power-loss recovery is not claimed.
+
+## 3. Driver: ordered boot installation and pinned owner selection
+
+The offline target boot list is:
+
+```json
+[
+  {"method":"session.manager","params":{}},
+  {"method":"web.start","params":{}},
+  {"method":"cred.serve","params":{}},
+  {"method":"orchestrator.start","params":{}},
+  {"method":"cron.start","params":{}}
+]
+```
+
+Manager precedes web, so `web-chat` becomes best-effort. No `mail.start` deliverer remains. Resolve any extra actual boot entry before approval rather than deleting it accidentally.
+
+The permanent `<root>/target-owner` exports `SAUSAGE_FLAKE=path:<root>/target-selector` itself, then execs the inspected loop with the exact world path. The selector's sole app is the pinned target package. `<root>/old-owner` similarly selects the **explicit old package**, never the remote default. These are permanent installation/launcher artifacts, **not migration code**. Their bytes/hashes are retained as manifest evidence; do not delete them with `tool/`.
+
+**Every later pane/cold restart uses `/home/surma/sau-384-cutover/target-owner`, with no caller environment required. Never invoke bare sausage-loop or the mutable gitea default for this world.** Keep this command in the operator's pane-restart procedure. Rehearsal cold-launched it in the same Herdr shell without ambient SAUSAGE_FLAKE and observed the target executable. Earlier cold rollback exercised the explicit old wrapper against a separate old-backup clone; pre-start abort gates also exercised it against the exactly restored original frozen fixture, never over new target work.
+
+`core.restart` now restarts the pinned reviewed host; it does **not** self-update from gitea. Future host updates require a separately approved package/closure/root/selector change. A remote push alone must never silently choose an old host against target state. Keep both GC roots through the rollback window; retire the old root only after an explicit decision, not tool deletion.
+
+**Expected approved live Nix side effect:** each controller run makes three `nix-store --query --roots` calls, one each for old package, target package and loop, immediately after rooting those paths and before freeze/resume of offline work. **Every resume repeats all three queries and their possible pruning**, even if the roots already exist. Monitor/check/arm/continue/verify alone do not make those roots queries. Those queries can prune stale auto-root registrations and stale temporary-root files in Nix state, even though they do not delete store packages. Surma's go must include that side effect (and the five root registrations including controller/Herdr); the earlier shared-store incident is disclosed in rework-1381.md. Rehearsal now uses only a private Nix DB; no live root query was rerun.
+
+## 4. Driver: started → launch in the same pane → fresh verifier
+
+The driver checks the pinned shell still owns the selected pane and no unexpected foreground process exists. It checkpoints **`started` before any launch attempt**, then submits the permanent target-owner path once via `herdr pane run` to the manifest's pane (live **w7:p6**). No shell reconstruction or in-world mediator action is needed during downtime.
+
+Readiness is event driven: arm a kernel inotify watch on `state/session/manager/registrations`, queue one `session.subscribe` command in the console, and await its unique registration file publication. The wait has **one absolute 120 s deadline**, from cron `src/types.rs` AddParams' existing registration/publication budget (also the earlier test_driver completion bound); it is not a periodic scan. Herdr/RPC calls use that existing finite bound. Missing registration or a failed post-start `cred.status`/subscribe/send produces the disk-independent failure/death signal and exits, keeping **`started`**. Never resume, restore or launch again automatically. Herdr wait-output is internally polling and is not used.
+
+### Forward completion after a post-start failure (Surma's independent terminal)
+
+1. Read checkpoint/monitor and inspect Herdr pane/native identities once. If target is not actually running, preserve all new work and obtain a narrow forward repair plan; this mode will refuse and **does not launch it**. Do not relaunch blindly, restore an old snapshot or hand-edit the checkpoint. If only registration/publication failed but exactly one target is running, use:
+
+```sh
+cd /home/surma/sau-384-cutover
+PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
+PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py continue operator.json
+```
+
+2. `continue` exclusively locks, binds the unchanged manifest, inspects that exact shell/pane, exactly one target host for this world and all offline target plugin hashes. It never freezes, applies, restores or launches. It then performs the **skipped exact RPCs**, through operator.json's gateway: `cred.status {}` must report locked and is durably saved as `driver.json.locked_cred_status`/`locked_cred:true` **before Surma unlocks**; `session.subscribe` uses id `cutover:<continuation_id>`, session `cutover-readiness`, method `web.session_event`, best_effort true; `session.send` uses session default, the **same** continuation_id, kind follow-up and exact continuation_text. It saves the receipt and reaches `awaiting_verification`. A recorded locked-startup proof is reused if a later retry occurs after UI unlock; absent that proof, unlocking first is a discrepancy, not permission to fake the check.
+3. If a method/reply fails again, keep target work, inspect the error and fix only that cause; repeat the **same continue command**, not migration/start. Stable subscribe/message IDs make this safe even when send committed and only its reply was lost. Both started and awaiting_verification are accepted. Tests injected each RPC failure and committed-then-lost send acknowledgement; one continuation run-start/consumption remained. The real 120 s registration timeout also recovered through this command without a second owner.
+4. `verify` still requires awaiting_verification. Run continue first when phase is started; there is no direct started→verified override. The fresh mediator can then verify/ask for unlock/release as §5; if locked cred prevents SSH signing, Surma performs the same approved local external commands. Live SSH/new-cred signing is not claimed tested. No hand-edited checkpoint or new continuation UUID is needed.
+
+Finally the driver publishes the manifest's **new explicit default verification message** with its stable UUID and records `awaiting_verification`. The manager, not the driver, runs it. This names the new mediator as verifier without claiming the cut default run was recovered. The test produced one run-start and mailbox-consumed pair for that message. Publication is not immediate model execution: in the last gate it waited **23.753 s to run-start / 24.530 s to consumption** behind initial reconciliation. Do not confuse that with publication latency or a completed startup scan.
+
+No automatic relaunch, whole-world restore or reapply is allowed after `started`. If launch/acknowledgement is lost, inspect the pane/checkpoint once before any repair. Do not launch a second owner blindly.
+
+## 5. New mediator: locked proof → Surma unlock → git → release web → delete tool
+
+The explicit new verification run reads this runbook and in-world ownership/cred state. While cred is locked, Surma checks the external manifest/checkpoint/migration evidence in his independent terminal; the new mediator can read it through approved SSH only after UI unlock if signing works. Surma can also perform the final local verify command, so no vault-key transfer or untested signer is required to finish. Keep ingress closed until checks pass:
+
+- one target manager and one runner ownership lock per running session; no old owners, mail deliverer or old orchestrator service;
+- cred **locked**, no vault/key transfer; grants/requests/settings preserved. Interrupted callers reuse their existing stable credential-request keys, not automatic retries of secret-backed mutations;
+- explicit target run-start without run-end is the only automatic cut-run recovery; old finished-looking history alone is neither settlement nor permission to resume;
+- durable event/mailbox catch-up and same-ID default/advisor POST replay evidence holds; only intended loss/conversion/schema/new-activity differences exist. Surface uncertain mail/tool outcomes;
+- scheduled jobs and non-target state match the inventory; real new mediator/runner activity is distinguished from migration changes;
+- design effective grace 5 s and separate EOF-stop 5 s remain; crash supervision is enabled. Deliberate core.stop/kill disables service registration and is **not** a crash-supervision test.
+
+Prior post-record live-size samples remain manager ack **0.6078 s**, registration catch-up **0.0128 s**, mail.send-to-default publication **0.5376 s** (integrated 0.5235 s), below existing orchestrator **3 s** / cred **10 s** caller budgets. Disclose integrated catch-up **12.0419 s** and the new initial-scan delay above. They are synthetic/warm-cache observations, not production latency guarantees. Prior real-core watch/upgrade/all-watch readiness, ~0.10% whole-host idle duty, detached error data and real cron warning/stop evidence remain linked in the result docs.
+
+Before any Git network work, record locked-startup proof, ask **Surma to unlock the target vault through the existing UI**, then check `cred.status` reports unlocked for that same target service. Wait here if Surma is unavailable; never transfer a key/password. **Only after that**, mediator/orchestrator activates the reviewed git artifact:
+
+```json
+{"method":"core.activate","params":{"plugin":"git","wasm":"/docs/symphony/artifacts/SAU-384/git-9cc9fa6f.wasm"}}
+```
+
+This is a proposed live operator step covered by the go, not a worker action. The permanent world artifact above must be staged/hash-checked from the selected bundle before go; the driver preflights its external `deferred_artifacts.git` copy too. Verify active git hash equals manifest `files["git.wasm"].sha256` before releasing Git callers. See `/plugins/git/source/{VAULT-HTTPS.md,VAULT-SSH.md}`; both require cred first, live and unlocked. No push is part of verification; **every** later push still needs a separate fresh Surma go. Final proof includes `git_activated_after_unlock`.
+
+Interim compatibility: observed live Git **f089b56d** still speaks new cred's explicit-grant fetch/item_get/public_key/sign contracts. Actual old-WASM + new-cred loopback SSH clone signed/authenticated successfully with a synthetic explicit grant after unlock; locked use returned vault_locked. Granted HTTPS reached new cred, reserved exactly one use, then the deliberately closed loopback port returned network_error with a capture (no successful HTTPS transfer claimed). Old grant-free shared SSH fails asking for a local key; old grant-free HTTPS retains its old config/anonymous path, not new anyone_may_use selection. Therefore do not schedule Git network work during the locked/old-git interval, do not introduce local key/config-secret workarounds, and surface any already-started caller's lock/missing-key error rather than blindly retrying it. Local git status/commit/authoritative-main merges remain available. New Git shared SSH clone was tested after real synthetic unlock/activation.
+
+Only now use **`orchestrator.release` for SAU-375** to release its approved commit. Inspect that method's current typed arguments; do not invent them. Verify published asset hashes and ingress. No worker publishes anything. The rework tested approved asset hashes in temporary output, **not an actual live release or physical browser**.
+
+The Surma unlock above is a prerequisite for using the existing approved signing grant for the final external SSH verification/deletion command (live signing interoperability is still unverified); it is not a second vault handover. Never ask for Surma's password or transfer the old vault key. Read-only gateway checks and checkpoint inspection do not justify bypassing the lock. If unlock is not available, remain `awaiting_verification`, keep git activation/ingress/release gated, and do not reconstruct or rerun the migration.
+
+Write external `verification.json` with evidence links and the required true checks named by the driver's `verify` mode: `one_manager`, `no_old_owner`, `locked_cred`, `non_target_preserved`, `losses_exact`, `catchup_and_replay`, `web_released_and_hashes`, `ingress_verified`, `git_activated_after_unlock`. They are operator attestations after the checks, not substitutes for them. From the external root, mediator runs the finite command below after target cred is unlocked and signing works; **Surma may instead run the identical command in his independent Citadel terminal**, avoiding any dependency on in-world SSH signing:
+
+```sh
+PYTHONDONTWRITEBYTECODE=1 /nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3 -B tool/driver.py verify operator.json verification.json 2>> tool-source-and-delete.log
+```
+
+It marks the saved plan/driver verified, logs source hashes/source-as-log, and **deletes the three migration executables and bytecode caches**. Confirm `tool-deletion.json` and absence once. Delete any other reconstructed executable/controller copies too. Retain only manifests, backup/inventory/loss/caller-audit records and logs as migration evidence, plus the separately reviewed target/permanent launcher artifacts. Rehearsal deleted all executed migration/test controllers before exporting evidence. If verification/deletion is interrupted, inspect the verified checkpoint and delete only remaining hash-matched tool copies; do not reconstruct and rerun migration effects.
+
+## Failure/rollback boundaries
+
+| Boundary | Exact recovery rule |
+|---|---|
+| Waiting for arm, including consumed-marker/save race | World untouched. Repeated arm is finite even with no reader. After the saved controller exits, `resume` accepts waiting_for_arm and the durable marker; attach a new `monitor` for that replacement before it continues. Or run abort (untouched) after obtaining the external controller lock. |
+| Armed/captured/owner_stopped/frozen; no target attempt | External operator can use `... driver.py resume operator.json` **only after the saved controller has exited**. It detaches a replacement and returns; Surma then runs the same foreground monitor command. The replacement waits for its own monitor receipt before effects. Same boot/identities/manifest/backup/plan must pass. No agent needs to exist. Both caller-killed and controller-killed resume were rehearsed. |
+| Interrupted planning/swap, no start attempt | Resume the same immutable backup/plan with before/after hashes. A caller/config after-image is accepted only if that exact saved operation approved it; third images refuse. Alternatively Surma runs the real driver's **abort** mode below: it restores every approved changed path from the immutable complete backup, verifies the full frozen inventory, and launches the explicit old-owner in the selected pane. It never restores an incomplete backup. If planning never completed, apply cannot have started: abort first discards only `migration/backup` (including its partial copy temporaries), retaining planning/inventory/log evidence, then checks space **before any checkpoint save**. If enough space is available, it verifies any saved inventory and relaunches the untouched old world. If a competing writer still fills the disk it refuses before writes/launch; Surma must free only identified, separately approved non-world data and repeat abort. Never delete a complete backup/plan or unrelated world data to make space. Restore is before/after-hash guarded, resumable, and writes main DBs before their old sidecars without opening them. Never partially start mixed contracts. A frozen audit discrepancy is an offline stop, not permission to improvise a conversion. |
+| `started` or later, even if boot seems failed | No automatic whole-world rollback/reapply/relaunch. New histories, mail, jobs, grants or external effects may exist. Preserve them. Use the forward-only continue command above for readiness/post-start RPC failure with one healthy target; otherwise plan a narrow forward repair or coordinated revert. |
+| Any plugin core.revert | Offline-swapped prev now means exactly pre-cut current (hash-tested), not pre-pre-cut. **Do not use blind individual core.revert across incompatible contracts.** Coordinate providers/callers and preserve new work; the session prerequisite below still applies. Later git activation uses normal core prev semantics. |
+| Session revert after target activity | Close gateway/ingress. While new manager is available, **unsubscribe web-chat**, verify removal, and prevent gateway re-registration while the old session binary runs. Never call web.start against that old manager or restore a registry backup that reintroduces it. Actual old-manager/gateway tests proved unsubscribe safe and ID-only re-key unsafe: the old binary ignores best_effort and replays history once. |
+| Host reboot / changed PID identity / unexpected survivor | Refuse saved native identities; re-inventory externally. Never signal a reused PID, blindly kill a group, or infer quiescence from a stop request. |
+
+### Exact external failure commands (Surma, not an in-world mediator)
+
+In the independent Citadel terminal, `cd /home/surma/sau-384-cutover`; read `driver.json`, monitor output, any available `failure.json`/`driver.log`, and the pane once. At ENOSPC those disk-backed diagnostics may be absent or truncated; the terminal failure/death signal is authoritative that the controller ended, not that rollback is safe. Commands use the pinned interpreter:
+
+```sh
+PY=/nix/store/fixvq6n42pmvy49yh6kfmpax1s3j8md2-python3-3.14.7-env/bin/python3
+# Same manifest/boot/owners, known exited controller, pre-start only:
+PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py resume operator.json
+# After resume acknowledges, attach its new foreground monitor before effects:
+PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py monitor operator.json
+# Or abandon pre-start attempt, restore coherent old bytes, submit old-owner once:
+PYTHONDONTWRITEBYTECODE=1 "$PY" -B tool/driver.py abort operator.json
+```
+
+Choose **one**, not both. Both use a nonblocking exclusive controller lock; do not delete the lock file or bypass it. Abort accepts waiting/armed/frozen/installed/restoring/restored only. It checkpoints `old_start_attempted` **before** pane launch, then `old_launched`. If interrupted during restoring/restored, repeat abort on the same manifest/checkpoints; if `old_start_attempted`/`old_launched`, inspect the pane instead of repeating it. Abort is forbidden after any target `started` attempt. Never call Kit.rollback by hand or relaunch a bare loop. If the pinned shell/pane changed, stop for explicit identity resolution; do not guess a replacement.
+
+After abort, Surma verifies exactly one old package owner in the pane and old world readiness before retry. Preserve the frozen evidence; the sole permitted deletion of backup evidence before verified success is the **reproducible incomplete pre-plan backup subtree** in the guarded abort path above. Complete backups are never deleted by abort. Every abort, including one with a completed plan, checks space before its first checkpoint save; a full disk is a deliberate pre-write refusal, not a failed checkpoint write. Abort rechecks the same measured space budget again immediately before old launch; if it refuses, free only identified/approved competing non-world data, not histories, jobs or complete backup evidence, then repeat abort on the same checkpoint. For a clean retry **only after old_launched and verification**, archive checkpoints (not owners/selectors/GC roots/tool):
+
+```sh
+# Pick a fresh, unused archive path and record it; this example is this run.
+ARCHIVE=attempt-before-retry
+PYTHONDONTWRITEBYTECODE=1 "$PY" -B -c 'import json; assert json.load(open("driver.json"))["phase"] == "old_launched"'
+mkdir -- "$ARCHIVE"
+for p in driver.json freeze.json arm.json controller-process.json migration driver.log failure.json failure-notification.json preflight-live.json monitor.json poststart-inspection.json; do
+  if [ -e "$p" ]; then mv -- "$p" "$ARCHIVE/"; fi
+done
+cp -- operator.json "$ARCHIVE/operator.json"
+```
+
+Re-identify the now-running old loop/host and pane shell; refresh old-work/caller guards (including any wake-up) and approval for the new resolved `operator.json`, retaining the stable continuation UUID if no target start was attempted. Then check/start/arm as §1. Do not merely edit a live checkpoint or delete the backup. For aborted_untouched, no old relaunch is necessary; archive only after verifying the original owners are still running. No such reset/archive is allowed for started/awaiting_verification/verified or unknown owners: retain new work and plan a forward repair.
+
+Independent plugin recovery/accounting/provider-cancellation follow-ups remain independent. Accepted first-update cuts and uncertain external effects are not silently solved by this kit, grace=5, or a successful rehearsal.

sdk/macros/src/lib.rs

@@ -153,6 +153,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
         let mut stop = quote!();
         let mut has_stop = false;
         let mut has_retired = false;
+        let mut has_superseded = false;
         for member in &mut item.items {
             let ImplItem::Fn(f) = member else { continue };
             let retired = f.attrs.iter().any(|a| {
@@ -161,6 +162,12 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
                     .last()
                     .is_some_and(|s| s.ident == "on_retired")
             });
+            let superseded = f.attrs.iter().any(|a| {
+                a.path()
+                    .segments
+                    .last()
+                    .is_some_and(|s| s.ident == "on_superseded")
+            });
             let stopping = f.attrs.iter().any(|a| {
                 a.path()
                     .segments
@@ -179,11 +186,16 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
                     "only one on_retired method is allowed",
                 ));
             }
+            if superseded && std::mem::replace(&mut has_superseded, true) {
+                return Err(syn::Error::new(
+                    f.span(),
+                    "only one on_superseded method is allowed",
+                ));
+            }
             f.attrs.retain(|a| {
-                !a.path()
-                    .segments
-                    .last()
-                    .is_some_and(|s| s.ident == "on_retired" || s.ident == "on_stop")
+                !a.path().segments.last().is_some_and(|s| {
+                    s.ident == "on_retired" || s.ident == "on_stop" || s.ident == "on_superseded"
+                })
             });
             documented(&f.attrs, f.span())?;
             if f.sig.inputs.is_empty()
@@ -224,7 +236,9 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
                 stop = quote!(fn stop(&self)->::std::pin::Pin<Box<dyn ::std::future::Future<Output=()> + '_>> {Box::pin(async move {let _=self.#name() #await_;})});
                 continue;
             }
-            let wire = if retired {
+            let wire = if superseded {
+                "rpc.superseded".to_string()
+            } else if retired {
                 "rpc.retired".to_string()
             } else {
                 name.unraw().to_string()
@@ -253,7 +267,7 @@ pub fn rpc(args: TokenStream, input: TokenStream) -> TokenStream {
             };
             let doc = docs(&f.attrs);
             arms.push(quote!(#wire => { fn params_type<T: ::sausage_plugin::Params>() {} params_type::<#p>(); let result=self.#name(#(#args),*) #await_ ?; Ok(::sausage_plugin::serde_json::to_value(result)?) }));
-            if !retired {
+            if !retired && !superseded {
                 entries.push(quote!(::sausage_plugin::method::<#p,#output>(&mut doc,#wire,#doc);));
             }
         }
@@ -328,6 +342,12 @@ pub fn import(input: TokenStream) -> TokenStream {
 pub fn on_retired(_: TokenStream, input: TokenStream) -> TokenStream {
     input
 }
+/// Mark the method that handles `rpc.superseded` at the start of upgrade grace.
+/// Calls still work here. Use `sausage_plugin::drain` to finish active work and exit.
+#[proc_macro_attribute]
+pub fn on_superseded(_: TokenStream, input: TokenStream) -> TokenStream {
+    input
+}
 /// Mark the hook that runs at stdin EOF without a live call transport.
 #[proc_macro_attribute]
 pub fn on_stop(_: TokenStream, input: TokenStream) -> TokenStream {

sdk/src/calls.rs

@@ -1,5 +1,19 @@
 //! Outgoing core calls and notification callbacks.
 use super::*;
+/// Removes a dropped call's local routing state, without cancelling host work.
+struct PendingCall(u64);
+impl Drop for PendingCall {
+    fn drop(&mut self) {
+        let pending = STATE.with(|s| s.borrow_mut().pending.remove(&self.0));
+        if let Some(mut pending) = pending {
+            // A lock may still be granted: release it when its late response arrives.
+            if let Some(lease) = pending.unlock.take() {
+                STATE.with(|s| s.borrow_mut().abandoned_locks.insert(self.0, lease));
+            }
+            // Drop user-owned callback captures outside the STATE borrow.
+        }
+    }
+}
 /// An outgoing request builder.
 #[must_use = "await the call to send it"]
 pub struct Call {
@@ -26,6 +40,21 @@ impl Call {
         self
     }
     /// Receive progress notifications before the result arrives.
+    /// Dropping the call future removes its local pending entry and callback;
+    /// later notifications and responses are ignored. It does not cancel host work.
+    ///
+    /// For `core.watch`, poll the call future alongside work in the handler, then
+    /// drop it when the handler returns. Do not put it in `background()` just to
+    /// keep the watch alive: background tasks keep the process alive, while the
+    /// watch only ends when the process ends. Dropping the future does not cancel
+    /// the host watch, but it lets an otherwise idle process exit normally. A
+    /// long-lived service should keep one watch per purpose for its lifetime,
+    /// rather than repeatedly opening and dropping watches on the host.
+    ///
+    /// Scan only after `armed`, and rescan on `changed`. Events may coalesce, but
+    /// are not rate-limited: watch the narrowest useful path to avoid rescanning
+    /// for unrelated writes (for example, watch a mailbox, not all session data).
+    /// Do not use endpoint connect retries to wait for file publication.
     pub fn on_notify(mut self, f: impl FnMut(&str, Value) + 'static) -> Self {
         self.notify = Some(Box::new(f));
         self
@@ -60,6 +89,7 @@ impl IntoFuture for Call {
                 );
                 Ok(id)
             })?;
+            let _pending = PendingCall(id);
             let (method, params) = if self.detach {
                 (
                     "core.call".to_string(),
@@ -69,7 +99,8 @@ impl IntoFuture for Call {
                 (self.method, self.params)
             };
             if let Err(e) = send(json!({"jsonrpc":"2.0","id":id,"method":method,"params":params})) {
-                STATE.with(|s| s.borrow_mut().pending.remove(&id));
+                let pending = STATE.with(|s| s.borrow_mut().pending.remove(&id));
+                drop(pending);
                 return Err(e);
             }
             rx.await
@@ -77,3 +108,154 @@ impl IntoFuture for Call {
         })
     }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::{STATE, State, block_on, call, clear_pending, lock, receive};
+    use crate::Result;
+    use futures::{FutureExt, future::LocalBoxFuture};
+    use serde_json::{Value, json};
+    use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Context};
+
+    fn activate() {
+        STATE.with(|s| {
+            *s.borrow_mut() = State {
+                active: true,
+                ..State::default()
+            };
+        });
+    }
+    fn start(
+        mut call: LocalBoxFuture<'static, Result<Value>>,
+    ) -> (u64, LocalBoxFuture<'static, Result<Value>>) {
+        assert!(
+            call.as_mut()
+                .poll(&mut Context::from_waker(std::task::Waker::noop()))
+                .is_pending()
+        );
+        (STATE.with(|s| s.borrow().next), call)
+    }
+    fn notification(id: u64, name: &str) {
+        receive(json!({"method":"core.notify","params":{"call":id,"name":name,"params":{}}}));
+    }
+    fn response(id: u64, result: Value) {
+        receive(json!({"id":id,"result":result}));
+    }
+
+    #[test]
+    fn dropped_call_removes_pending_and_callback() {
+        activate();
+        let events = Rc::new(RefCell::new(Vec::new()));
+        let captured = events.clone();
+        let (id, future) = start(
+            call("core.watch", json!({"path":"/mailbox"}))
+                .on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
+                .into_future(),
+        );
+        notification(id, "armed");
+        assert_eq!(&*events.borrow(), &["armed"]);
+        assert_eq!(Rc::strong_count(&events), 2);
+        drop(future);
+        STATE.with(|s| {
+            assert!(s.borrow().pending.is_empty());
+            assert!(s.borrow().abandoned_locks.is_empty());
+        });
+        assert_eq!(Rc::strong_count(&events), 1);
+        notification(id, "changed");
+        response(id, Value::Null);
+        receive(json!({"id":id,"error":{"code":-32000,"message":"late error"}}));
+        assert_eq!(&*events.borrow(), &["armed"]);
+    }
+
+    #[test]
+    fn kept_watch_delivers_until_result() {
+        activate();
+        let events = Rc::new(RefCell::new(Vec::new()));
+        let captured = events.clone();
+        let (id, future) = start(
+            call("core.watch", json!({"path":"/mailbox"}))
+                .on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
+                .into_future(),
+        );
+        notification(id, "armed");
+        notification(id, "changed");
+        notification(id, "changed");
+        response(id, json!({"done":true}));
+        assert_eq!(block_on(future).unwrap(), json!({"done":true}));
+        notification(id, "late");
+        assert_eq!(&*events.borrow(), &["armed", "changed", "changed"]);
+        STATE.with(|s| assert!(s.borrow().pending.is_empty()));
+        assert_eq!(Rc::strong_count(&events), 1);
+    }
+
+    #[test]
+    fn callback_can_drop_its_own_call_without_being_restored() {
+        activate();
+        let holder = Rc::new(RefCell::new(None));
+        let captured = holder.clone();
+        let (id, future) = start(
+            call("core.watch", json!({"path":"/mailbox"}))
+                .on_notify(move |_, _| {
+                    // Panics on a second invocation; the first must remove the entry.
+                    drop(captured.borrow_mut().take().expect("live call"));
+                })
+                .into_future(),
+        );
+        *holder.borrow_mut() = Some(future);
+        notification(id, "armed");
+        STATE.with(|s| assert!(s.borrow().pending.is_empty()));
+        notification(id, "changed");
+        response(id, Value::Null);
+        assert_eq!(Rc::strong_count(&holder), 1);
+    }
+
+    #[test]
+    fn eof_cleanup_can_drop_callback_owned_futures() {
+        activate();
+        let holder = Rc::new(RefCell::new(None));
+        let weak = Rc::downgrade(&holder);
+        let captured = holder.clone();
+        let (_, future) = start(
+            call("core.watch", json!({"path":"/mailbox"}))
+                .on_notify(move |_, _| {
+                    let _ = &captured;
+                })
+                .into_future(),
+        );
+        *holder.borrow_mut() = Some(future);
+        drop(holder);
+        STATE.with(|s| s.borrow_mut().stopping = true);
+        clear_pending();
+        assert!(weak.upgrade().is_none());
+        STATE.with(|s| assert!(s.borrow().pending.is_empty()));
+    }
+
+    #[test]
+    fn unpolled_call_never_registers() {
+        activate();
+        drop(call("core.watch", json!({"path":"/mailbox"})).into_future());
+        STATE.with(|s| {
+            assert!(s.borrow().pending.is_empty());
+            assert_eq!(s.borrow().next, 0);
+        });
+    }
+
+    #[test]
+    fn dropped_lock_keeps_only_late_grant_cleanup() {
+        activate();
+        let (id, future) = start(
+            lock("/late")
+                .into_future()
+                .map(|r| r.map(|_| Value::Null))
+                .boxed_local(),
+        );
+        drop(future);
+        STATE.with(|s| {
+            assert!(s.borrow().pending.is_empty());
+            assert_eq!(s.borrow().abandoned_locks.len(), 1);
+        });
+        notification(id, "ignored");
+        response(id, json!({"held":true}));
+        STATE.with(|s| assert!(s.borrow().abandoned_locks.is_empty()));
+    }
+}

sdk/src/endpoints.rs

@@ -65,7 +65,7 @@ impl<T: serde::de::DeserializeOwned + 'static> IntoFuture for EndpointCall<T> {
     }
 }
 
-/// Publish a declared interface and keep its listener alive until stdin EOF.
+/// Publish a declared interface and keep its listener alive until stdin EOF or [`drain`].
 #[cfg(target_os = "wasi")]
 pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Result<()> {
     use wstd::iter::AsyncIterator;
@@ -80,7 +80,7 @@ pub async fn listen<T: Interface>(path: impl AsRef<Path>, value: Rc<T>) -> Resul
         while let Some(Ok(stream)) = incoming.next().await {
             let value = value.clone();
             let name = name.clone();
-            crate::transport::endpoint_task(async move {
+            background(async move {
                 let _ = serve_connection(stream, value, name).await;
             });
         }
@@ -211,7 +211,11 @@ async fn serve_connection<T: Interface>(
     use futures::{FutureExt, StreamExt};
     let (mut reader, mut writer) = stream.split();
     let mut lines = crate::lines::Lines::default();
-    while let Some(request) = lines.rpc(&mut reader).await? {
+    while let Some(request) = crate::transport::until_drained(lines.rpc(&mut reader))
+        .await
+        .transpose()?
+        .flatten()
+    {
         let method = request["method"].as_str().unwrap_or_default();
         let (tx, mut rx) = futures::channel::mpsc::unbounded();
         let ctx = Ctx {

sdk/src/lib.rs

@@ -356,13 +356,20 @@
 //!
 //! The interface check covers calls, not state. Old and new processes share files in `/state`.
 //! `core.revert` restores code, not state. Preserve compatibility or migrate the state.
-//! `#[rpc(grace_seconds = N)]` declares the grace period for old processes. The default is 60 seconds.
+//! Core gives every old process 5 seconds of upgrade grace, then closes stdin and allows
+//! a separate 5-second stop budget before killing it. Discovery-only processes are excluded.
+//! `#[rpc(grace_seconds = N)]` still emits legacy `x-sausage-grace` metadata, but core ignores it.
 //! New calls use the new version immediately. Old processes can still use old imports during their grace period.
+//! [`macro@on_superseded`] handles the notice at grace start, with normal calls still available.
+//! It is opt-in: without a hook the process continues as before until core stops it.
+//! Call [`drain`] in this hook to close SDK endpoint listeners and idle connections,
+//! let active handlers and background tasks finish, and exit without waiting for grace.
+//! Custom listeners and long-lived background tasks must be ended by the plugin itself.
 //! [`macro@on_retired`] handles `rpc.retired` after the last process of an old version ends.
 //! This hook is best effort. If the host stops first, the hook does not run.
 //! New processes must handle unmigrated state. Check a migration marker and migrate if it is absent.
 //! See section 3.7 of `/proc/core/PROTOCOL.md`.
-//! Declare at most one stop hook and one retirement hook per interface.
+//! Declare at most one hook of each kind per interface.
 //!
 //! ```no_run
 //! use sausage_plugin::{rpc, schema, state_dir, write_atomic, Result};
@@ -374,8 +381,21 @@
 //!     /// The SHA-256 of the previous WASM file.
 //!     previous: String,
 //! }
+//! /// The version now receiving new calls.
+//! #[schema]
+//! struct SupersededParams {
+//!     /// The SHA-256 of the newly active WASM file.
+//!     current: String,
+//! }
 //! #[rpc]
 //! impl Migrator {
+//!     /// Finish current work and exit early during upgrade grace.
+//!     #[sausage_plugin::on_superseded]
+//!     fn superseded(&self, p: SupersededParams) -> Result<()> {
+//!         eprintln!("replaced by {}", p.current);
+//!         sausage_plugin::drain();
+//!         Ok(())
+//!     }
 //!     /// Record a retired version. Normal methods must also handle missing markers.
 //!     #[sausage_plugin::on_retired]
 //!     fn retired(&self, p: RetiredParams) -> Result<()> {
@@ -401,6 +421,6 @@ pub use helpers::*;
 pub use imports::*;
 #[doc(hidden)]
 pub use inventory;
-pub use sausage_plugin_macros::{import, on_retired, on_stop, rpc, schema, test};
+pub use sausage_plugin_macros::{import, on_retired, on_stop, on_superseded, rpc, schema, test};
 pub use transport::*;
 pub use {schemars, serde, serde_json};

sdk/src/transport.rs

@@ -96,8 +96,11 @@ struct State {
     active: bool,
     endpoints: HashMap<std::any::TypeId, String>,
     stopping: bool,
+    draining: bool,
     next: u64,
     pending: HashMap<u64, Pending>,
+    // Only leases, never callbacks or response senders, survive dropped lock acquisitions.
+    abandoned_locks: HashMap<u64, std::rc::Rc<Lease>>,
     #[cfg(target_os = "wasi")]
     tasks: usize,
     #[cfg(target_os = "wasi")]
@@ -108,6 +111,18 @@ struct State {
     listeners: HashMap<u64, futures::future::AbortHandle>,
 }
 thread_local! { static STATE: RefCell<State> = RefCell::new(State::default()); }
+#[cfg(any(target_os = "wasi", test))]
+fn clear_pending() {
+    let pending = STATE.with(|s| {
+        let mut s = s.borrow_mut();
+        (
+            std::mem::take(&mut s.pending),
+            std::mem::take(&mut s.abandoned_locks),
+        )
+    });
+    // Callback captures may own other call futures whose drop guards use STATE.
+    drop(pending);
+}
 pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
     STATE
         .with(|s| {
@@ -125,17 +140,48 @@ pub(crate) fn endpoint_name<T: 'static>() -> Result<String> {
 }
 #[cfg(target_os = "wasi")]
 pub(crate) fn endpoint_task(future: impl Future<Output = ()> + 'static) {
+    background(async move {
+        let _ = until_drained(future).await;
+    });
+}
+/// Cancel an accept/read wait when draining, but never an active handler.
+#[cfg(target_os = "wasi")]
+pub(crate) async fn until_drained<F: Future>(future: F) -> Option<F::Output> {
     let (handle, registration) = futures::future::AbortHandle::new_pair();
     let id = STATE.with(|s| {
         let mut s = s.borrow_mut();
+        if s.draining {
+            return None;
+        }
         s.next += 1;
         let id = s.next;
         s.listeners.insert(id, handle);
-        id
-    });
-    background(async move {
-        let _ = futures::future::Abortable::new(future, registration).await;
-        STATE.with(|s| s.borrow_mut().listeners.remove(&id));
+        Some(id)
+    })?;
+    let result = futures::future::Abortable::new(future, registration)
+        .await
+        .ok();
+    STATE.with(|s| s.borrow_mut().listeners.remove(&id));
+    result
+}
+/// Stop accepting SDK endpoint work, then exit when handlers and background tasks finish.
+///
+/// Calls and notifications still work. This does not run the EOF stop hook or cancel
+/// background tasks: services with their own listeners/tasks must arrange for those
+/// to finish too. Repeated calls are harmless. Core's upgrade grace remains the bound.
+pub fn drain() {
+    STATE.with(|s| {
+        let mut s = s.borrow_mut();
+        s.draining = true;
+        #[cfg(target_os = "wasi")]
+        {
+            for (_, listener) in s.listeners.drain() {
+                listener.abort();
+            }
+            if let Some(w) = s.wake.take() {
+                w.wake();
+            }
+        }
     });
 }
 #[cfg(target_os = "wasi")]
@@ -192,16 +238,16 @@ pub fn test_run<F: Future>(f: F) -> F::Output {
             STATE.with(|s| {
                 let mut s = s.borrow_mut();
                 s.stopping = true;
-                s.pending.clear();
             });
+            clear_pending();
         });
         let result = f.await;
         drop(reader);
         STATE.with(|s| {
             let mut s = s.borrow_mut();
             s.active = false;
-            s.pending.clear();
         });
+        clear_pending();
         result
     })
 }
@@ -274,7 +320,16 @@ pub fn receive(value: Value) {
         }
         let _ = p.tx.send(result);
     } else {
-        eprintln!("note: response with unknown ID {}", value["id"]);
+        let lease = value["id"]
+            .as_u64()
+            .and_then(|id| STATE.with(|s| s.borrow_mut().abandoned_locks.remove(&id)));
+        if let Some(lease) = lease {
+            lease
+                .held
+                .set(value.get("error").is_none() && value["result"]["held"] == true);
+        } else {
+            eprintln!("note: response with unknown ID {}", value["id"]);
+        }
     }
 }
 
@@ -324,11 +379,11 @@ async fn serve<T: Interface>(value: T) {
                 STATE.with(|s| {
                     let mut s = s.borrow_mut();
                     s.stopping = true;
-                    s.pending.clear();
                     for (_, listener) in s.listeners.drain() {
                         listener.abort();
                     }
                 });
+                clear_pending();
                 value.stop().await;
                 break;
             }
@@ -379,8 +434,8 @@ async fn serve<T: Interface>(value: T) {
     STATE.with(|s| {
         let mut s = s.borrow_mut();
         s.active = false;
-        s.pending.clear();
     });
+    clear_pending();
     let _ = std::io::stdout().flush();
     std::process::exit(0);
 }

sdk/tests/plugins/probe/src/main.rs

@@ -2,6 +2,12 @@
 use sausage_plugin::serde_json::{Value, json};
 use sausage_plugin::{Ctx, Result, call, lock, rpc, schema};
 struct Probe;
+/// A virtual path to observe.
+#[schema]
+struct WatchPath {
+    /// Directory or file, not necessarily present yet.
+    path: String,
+}
 /// The values to add.
 #[schema]
 struct Add {
@@ -16,6 +22,12 @@ struct Retired {
     /// The previous hash.
     previous: String,
 }
+/// The replacement version.
+#[schema]
+struct Superseded {
+    /// The current hash.
+    current: String,
+}
 /// A large payload.
 #[schema]
 struct Echo {
@@ -24,6 +36,98 @@ struct Echo {
 }
 #[rpc(grace_seconds = 12,endpoints(control=Control))]
 impl Probe {
+    /// Scan only after the watch is armed; keep the call open until process death.
+    async fn watch(&self, p: WatchPath) -> Result<()> {
+        let path = p.path.clone();
+        call("core.watch", json!({"path":p.path}))
+            .on_notify(move |name, _| {
+                if name == "armed" && path == "/self" {
+                    std::fs::write("/self/owned", "visible via proc").unwrap();
+                }
+                let mut entries: Vec<_> = std::fs::read_dir(&path)
+                    .into_iter()
+                    .flatten()
+                    .filter_map(std::result::Result::ok)
+                    .map(|e| e.file_name().into_string().unwrap())
+                    .collect();
+                entries.sort();
+                let text = std::fs::read_to_string(format!("{path}/before")).ok();
+                Ctx::default()
+                    .notify(
+                        name,
+                        json!({"pid":sausage_plugin::pid(), "entries":entries, "text":text}),
+                    )
+                    .unwrap();
+            })
+            .await?;
+        Ok(())
+    }
+
+    /// Poll the watch in the handler, then return idle without a background task.
+    async fn watch_once(&self, p: WatchPath) -> Result<()> {
+        use std::{
+            cell::RefCell,
+            rc::Rc,
+            task::{Poll, Waker},
+        };
+        let state = Rc::new(RefCell::new((false, None::<Waker>)));
+        let received = state.clone();
+        let mut watch = std::pin::pin!(std::future::IntoFuture::into_future(
+            call("core.watch", json!({"path":p.path})).on_notify(move |name, _| {
+                if name == "armed" {
+                    let mut state = received.borrow_mut();
+                    state.0 = true;
+                    if let Some(waker) = state.1.take() {
+                        waker.wake();
+                    }
+                }
+            })
+        ));
+        std::future::poll_fn(|cx| {
+            if let Poll::Ready(result) = watch.as_mut().poll(cx) {
+                return Poll::Ready(result.map(|_| ()));
+            }
+            let mut state = state.borrow_mut();
+            state.1 = Some(cx.waker().clone());
+            if state.0 {
+                Poll::Ready(Ok(()))
+            } else {
+                Poll::Pending
+            }
+        })
+        .await
+    }
+
+    /// Drop one watch while another still delivers, in a process that remains alive.
+    async fn drop_watch(&self) -> Result<Vec<String>> {
+        use std::{cell::RefCell, future::IntoFuture, rc::Rc, task::Poll};
+        let events = Rc::new(RefCell::new(Vec::new()));
+        let captured = events.clone();
+        let mut dropped = call("core.watch", json!({"path":"/dropped"}))
+            .on_notify(move |_, _| captured.borrow_mut().push("dropped".to_owned()))
+            .into_future();
+        std::future::poll_fn(|cx| {
+            assert!(dropped.as_mut().poll(cx).is_pending());
+            Poll::Ready(())
+        })
+        .await;
+        drop(dropped);
+        let captured = events.clone();
+        let mut kept = call("core.watch", json!({"path":"/kept"}))
+            .on_notify(move |name, _| captured.borrow_mut().push(name.to_owned()))
+            .into_future();
+        std::future::poll_fn(|cx| {
+            assert!(kept.as_mut().poll(cx).is_pending());
+            Poll::Ready(())
+        })
+        .await;
+        // The harness sends late traffic and kept-watch events before answering.
+        call("other.barrier", json!({})).await?;
+        drop(kept);
+        let result = events.borrow().clone();
+        Ok(result)
+    }
+
     /// Publish a control endpoint until EOF.
     async fn serve(&self) -> Result<()> {
         sausage_plugin::listen("/endpoint", std::rc::Rc::new(Control)).await
@@ -54,6 +158,11 @@ impl Probe {
         sausage_plugin::listen("/state/control", std::rc::Rc::new(Control)).await?;
         Ok(sausage_plugin::pid())
     }
+    /// Keep an idle listener and ownership lock until opt-in drain ends this process.
+    async fn drain_service(&self) -> Result<String> {
+        lock("/drain-owner").await?.keep();
+        self.service().await
+    }
     /// Add two numbers.
     ///
     /// The result is their sum.
@@ -111,6 +220,15 @@ impl Probe {
     fn retired(&self, p: Retired) -> Result<String> {
         Ok(p.previous)
     }
+    /// Drain on replacement while calls are still available.
+    #[sausage_plugin::on_superseded]
+    async fn superseded(&self, p: Superseded) -> Result<()> {
+        call("other.superseded", json!({"current":p.current})).await?;
+        sausage_plugin::drain();
+        sausage_plugin::drain();
+        Ctx::default().notify("draining", json!({}))?;
+        Ok(())
+    }
     /// Wait until EOF, then run the stop hook.
     async fn wait(&self) -> Result<String> {
         call("other.wait", json!({})).await?;
@@ -155,6 +273,7 @@ impl Probe {
     #[sausage_plugin::on_stop]
     async fn stop(&self) {
         std::fs::write("/stopped", b"yes").unwrap();
+        std::fs::write(format!("/stopped-{}", sausage_plugin::pid()), b"yes").unwrap();
         Ctx::default().notify("stopped", json!({})).unwrap();
         assert!(call("other.no", json!({})).await.is_err());
     }
@@ -193,6 +312,11 @@ impl Control {
         ctx.notify("endpoint-progress", json!({"value":7}))?;
         Ok(7)
     }
+    /// Finish an in-flight call after draining begins.
+    async fn wait(&self) -> Result<String> {
+        call("other.inflight", json!({})).await?;
+        Ok("finished".into())
+    }
     /// End the endpoint process after the response.
     fn stop(&self, ctx: &Ctx) -> Result<()> {
         ctx.exit_after_response();

sdk/tests/ui/superseded.rs

@@ -0,0 +1,11 @@
+struct Supersedes;
+#[sausage_plugin::rpc]
+impl Supersedes {
+    /// First superseded hook.
+    #[sausage_plugin::on_superseded]
+    fn a(&self) -> sausage_plugin::Result<()> { Ok(()) }
+    /// Second superseded hook.
+    #[sausage_plugin::on_superseded]
+    fn b(&self) -> sausage_plugin::Result<()> { Ok(()) }
+}
+fn main() {}

sdk/tests/ui/superseded.stderr

@@ -0,0 +1,5 @@
+error: only one on_superseded method is allowed
+ --> tests/ui/superseded.rs:7:5
+  |
+7 |     /// Second superseded hook.
+  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^

sdk/tests/wasi.rs

@@ -108,6 +108,34 @@ fn protocol() {
         }
         p.finishes();
     }
+    // A handler-owned watch does not keep an idle process alive after return,
+    // even though the host never answers the watch and stdin remains open.
+    let mut p = start(&wasm, &dir);
+    p.request("watch_once", json!({"path":"/mailbox"}));
+    let watch = p.read();
+    assert_eq!(watch["method"], "core.watch");
+    p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":"armed","params":{}}}));
+    assert_eq!(p.read()["result"], Value::Null);
+    p.finishes();
+    // Keep the process alive after dropping a call: late traffic is ignored,
+    // while the kept watch continues to deliver every notification.
+    let mut p = start(&wasm, &dir);
+    p.request("drop_watch", json!({}));
+    let dropped = p.read();
+    let kept = p.read();
+    let barrier = p.read();
+    assert_eq!(dropped["params"]["path"], "/dropped");
+    assert_eq!(kept["params"]["path"], "/kept");
+    assert_eq!(barrier["method"], "other.barrier");
+    for watch in [&dropped, &kept] {
+        for name in ["armed", "changed", "changed"] {
+            p.send(json!({"jsonrpc":"2.0","method":"core.notify","params":{"call":watch["id"],"name":name,"params":{}}}));
+        }
+    }
+    p.answer(&dropped, Value::Null);
+    p.answer(&barrier, json!({}));
+    assert_eq!(p.read()["result"], json!(["armed", "changed", "changed"]));
+    p.finishes();
     let mut p = start(&wasm, &dir);
     p.request("rpc.discover", json!({}));
     let d = p.read();
@@ -201,6 +229,103 @@ fn protocol() {
     assert_eq!(outgoing["method"], "other.background");
     p.answer(&outgoing, json!({}));
     assert!(p.child.wait().unwrap().success());
+    // Replacement is a notification, not EOF. Drain keeps both endpoint handlers
+    // and background calls alive, while closing idle connections and the listener.
+    assert!(
+        !d["result"]["methods"]
+            .as_array()
+            .unwrap()
+            .iter()
+            .any(|m| m["name"] == "rpc.superseded" || m["name"] == "superseded")
+    );
+    std::fs::remove_file(dir.join("stopped")).unwrap();
+    let mut provider = start(&wasm, &dir);
+    provider.request("serve", json!({}));
+    assert_eq!(provider.read()["result"], Value::Null);
+    provider.request("start", json!({}));
+    let first = provider.read();
+    let second = provider.read();
+    let background = if first["method"] == "other.background" {
+        first
+    } else {
+        second
+    };
+    assert_eq!(background["method"], "other.background");
+    let address = std::fs::read_to_string(dir.join("endpoint"))
+        .unwrap()
+        .split_whitespace()
+        .next()
+        .unwrap()
+        .trim_start_matches("tcp:")
+        .to_owned();
+    let mut idle = std::net::TcpStream::connect(&address).unwrap();
+    idle.set_read_timeout(Some(std::time::Duration::from_secs(4)))
+        .unwrap();
+    writeln!(
+        idle,
+        "{}",
+        json!({"jsonrpc":"2.0","id":0,"method":"rpc.endpoint","params":{}})
+    )
+    .unwrap();
+    let mut idle = BufReader::new(idle);
+    let mut line = String::new();
+    idle.read_line(&mut line).unwrap();
+    assert_eq!(
+        sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["id"],
+        0
+    );
+    let mut active = std::net::TcpStream::connect(&address).unwrap();
+    active
+        .set_read_timeout(Some(std::time::Duration::from_secs(4)))
+        .unwrap();
+    writeln!(
+        active,
+        "{}",
+        json!({"jsonrpc":"2.0","id":1,"method":"wait","params":{}})
+    )
+    .unwrap();
+    let inflight = provider.read();
+    assert_eq!(inflight["method"], "other.inflight");
+    provider
+        .send(json!({"jsonrpc":"2.0","method":"rpc.superseded","params":{"current":"new-hash"}}));
+    let hook = provider.read();
+    assert_eq!(hook["method"], "other.superseded");
+    assert_eq!(hook["params"]["current"], "new-hash");
+    provider.answer(&hook, json!({}));
+    assert_eq!(provider.read()["params"]["name"], "draining");
+    line.clear();
+    assert_eq!(idle.read_line(&mut line).unwrap(), 0);
+    // drain() wakes independent accept/read cancellations. Idle EOF does not
+    // order listener destruction: TCP may still complete a handshake, but that
+    // connection must close without serving work. Do not retry the connect.
+    if let Ok(mut late) = std::net::TcpStream::connect(&address) {
+        eprintln!("TCP handshake raced listener cancellation; waiting for close");
+        // Reuse the four-second socket bound used by idle/active above.
+        late.set_read_timeout(Some(std::time::Duration::from_secs(4)))
+            .unwrap();
+        let closed = std::io::Read::read(&mut late, &mut [0]);
+        assert!(
+            matches!(closed, Ok(0))
+                || matches!(&closed, Err(e) if e.kind() == std::io::ErrorKind::ConnectionReset),
+            "late connection survived drain: {closed:?}"
+        );
+    }
+    assert!(provider.child.try_wait().unwrap().is_none());
+    provider.answer(&inflight, json!({}));
+    line.clear();
+    BufReader::new(active).read_line(&mut line).unwrap();
+    assert_eq!(
+        sausage_plugin::serde_json::from_str::<Value>(&line).unwrap()["result"],
+        "finished"
+    );
+    assert!(provider.child.try_wait().unwrap().is_none());
+    provider.answer(&background, json!({}));
+    provider.finishes();
+    assert!(std::net::TcpStream::connect(&address).is_err());
+    assert!(
+        !dir.join("stopped").exists(),
+        "drain is not the EOF stop hook"
+    );
     let provider_doc = &d["result"];
     let imports = target.join("imports");
     std::fs::create_dir_all(&imports).unwrap();

vfs/src/lib.rs

@@ -5,6 +5,7 @@ pub mod generated;
 pub mod host;
 pub mod memory;
 pub mod mounts;
+pub mod observe;
 pub mod process;
 mod readonly;
 mod streams;

vfs/src/observe.rs

@@ -0,0 +1,615 @@
+//! Observes successful namespace and open-file mutations, even if their caller is cancelled.
+use crate::*;
+use std::{
+    collections::BTreeMap,
+    sync::{Mutex, Weak},
+};
+use tokio::sync::{OwnedRwLockReadGuard, OwnedRwLockWriteGuard, RwLock};
+
+type Namespace = Mutex<BTreeMap<String, Weak<RwLock<()>>>>;
+enum Held {
+    Read { _guard: OwnedRwLockReadGuard<()> },
+    Write { _guard: OwnedRwLockWriteGuard<()> },
+}
+
+type TrackedPath = Mutex<Option<String>>;
+type Path = Arc<TrackedPath>;
+type Callback = dyn Fn(&str) + Send + Sync;
+/// Shared observation for mount namespaces. Paths supplied to the callback are virtual paths.
+/// Callbacks run after filesystem locks have been released; they must not block.
+#[derive(Clone)]
+pub struct Observer {
+    changed: Arc<Callback>,
+    paths: Arc<Mutex<Vec<Weak<TrackedPath>>>>,
+    // Only open-registration versus rename/remove needs ordering. Lock the affected
+    // subtrees, not all IO: a blocked open must not stall unrelated namespaces.
+    namespace: Arc<Namespace>,
+}
+impl Observer {
+    pub fn new(changed: impl Fn(&str) + Send + Sync + 'static) -> Self {
+        Self {
+            changed: Arc::new(changed),
+            paths: Arc::default(),
+            namespace: Arc::default(),
+        }
+    }
+    /// Wrap a layer at its canonical namespace prefix (e.g. `/run`).
+    pub fn layer(&self, layer: Arc<dyn Layer>, prefix: &str, process: Process) -> Arc<dyn Layer> {
+        Arc::new(Observed {
+            layer,
+            prefix: prefix.trim_end_matches('/').into(),
+            observer: self.clone(),
+            process,
+        })
+    }
+    /// Opens share their path and ancestors; rename/remove exclusively hold their
+    /// endpoints, sharing ancestors. Sorted acquisition avoids cross-rename deadlocks.
+    /// No descriptor IO or notification holds these guards.
+    async fn order(&self, endpoints: &[(&str, bool)]) -> Vec<Held> {
+        let mut modes = BTreeMap::new();
+        for &(path, exclusive) in endpoints {
+            modes
+                .entry(path.to_owned())
+                .and_modify(|mode| *mode |= exclusive)
+                .or_insert(exclusive);
+            let mut ancestor = path;
+            while let Some((parent, _)) = ancestor.rsplit_once('/') {
+                let parent = if parent.is_empty() { "/" } else { parent };
+                modes.entry(parent.to_owned()).or_insert(false);
+                if parent == "/" {
+                    break;
+                }
+                ancestor = parent;
+            }
+        }
+        let locks: Vec<_> = {
+            let mut namespace = self.namespace.lock().unwrap();
+            namespace.retain(|_, lock| lock.strong_count() != 0);
+            modes
+                .into_iter()
+                .map(|(path, exclusive)| {
+                    let lock = namespace.entry(path).or_default();
+                    let strong = lock.upgrade().unwrap_or_else(|| {
+                        let strong = Arc::new(RwLock::new(()));
+                        *lock = Arc::downgrade(&strong);
+                        strong
+                    });
+                    (strong, exclusive)
+                })
+                .collect()
+        };
+        let mut held = Vec::new();
+        for (lock, exclusive) in locks {
+            held.push(if exclusive {
+                Held::Write {
+                    _guard: lock.write_owned().await,
+                }
+            } else {
+                Held::Read {
+                    _guard: lock.read_owned().await,
+                }
+            });
+        }
+        held
+    }
+    fn path(&self, path: String) -> Path {
+        let path = Arc::new(Mutex::new(Some(path)));
+        let mut paths = self.paths.lock().unwrap();
+        paths.retain(|p| p.strong_count() != 0);
+        paths.push(Arc::downgrade(&path));
+        path
+    }
+    fn moved(&self, from: &str, to: Option<&str>) {
+        let mut paths = self.paths.lock().unwrap();
+        paths.retain(|p| {
+            let Some(p) = p.upgrade() else { return false };
+            let mut path = p.lock().unwrap();
+            if let Some(old) = path.as_deref() {
+                if under(old, from) {
+                    *path = to.map(|to| format!("{to}{}", &old[from.len()..]));
+                } else if to.is_some_and(|to| under(old, to)) {
+                    // A replaced destination's old descriptor no longer names a visible file.
+                    *path = None;
+                }
+            }
+            true
+        });
+    }
+}
+/// Component-aware descendant matching, including the path itself.
+pub fn under(path: &str, base: &str) -> bool {
+    base == "/" || path == base || path.strip_prefix(base).is_some_and(|s| s.starts_with('/'))
+}
+struct Observed {
+    layer: Arc<dyn Layer>,
+    prefix: String,
+    observer: Observer,
+    process: Process,
+}
+impl Observed {
+    fn path(&self, path: &str) -> String {
+        let path = format!("{}{}", self.prefix, path.trim_end_matches('/'));
+        if path.is_empty() { "/".into() } else { path }
+    }
+}
+struct Handle {
+    file: Arc<dyn File>,
+    path: Path,
+    observer: Observer,
+    process: Process,
+}
+impl Handle {
+    async fn mutate<T: Send + 'static>(
+        &self,
+        f: impl Future<Output = Result<T>> + Send + 'static,
+    ) -> Result<T> {
+        let (path, observer) = (self.path.clone(), self.observer.clone());
+        self.process
+            .operation(async move {
+                let result = f.await?;
+                let path = path.lock().unwrap().clone();
+                if let Some(path) = path {
+                    (observer.changed)(&path);
+                }
+                Ok(result)
+            })
+            .await
+    }
+}
+#[async_trait]
+impl Layer for Observed {
+    async fn stat(&self, path: &str, follow: bool) -> Result<Stat> {
+        self.layer.stat(path, follow).await
+    }
+    async fn readlink(&self, path: &str) -> Result<String> {
+        self.layer.readlink(path).await
+    }
+    async fn list(&self, path: &str) -> Result<Vec<Entry>> {
+        self.layer.list(path).await
+    }
+    async fn open(&self, path: &str, options: Open) -> Result<Arc<dyn File>> {
+        let (layer, local, path, observer, process) = (
+            self.layer.clone(),
+            path.to_owned(),
+            self.path(path),
+            self.observer.clone(),
+            self.process.clone(),
+        );
+        self.process
+            .operation(async move {
+                let order = observer.order(&[(&path, false)]).await;
+                let file = layer.open(&local, options).await?;
+                let handle = Arc::new(Handle {
+                    file,
+                    path: observer.path(path.clone()),
+                    observer: observer.clone(),
+                    process,
+                });
+                drop(order);
+                if options.create || options.truncate {
+                    (observer.changed)(&path);
+                }
+                Ok(handle as Arc<dyn File>)
+            })
+            .await
+    }
+    async fn mkdir(&self, path: &str) -> Result<()> {
+        let (layer, local, path, observer) = (
+            self.layer.clone(),
+            path.to_owned(),
+            self.path(path),
+            self.observer.clone(),
+        );
+        self.process
+            .operation(async move {
+                layer.mkdir(&local).await?;
+                (observer.changed)(&path);
+                Ok(())
+            })
+            .await
+    }
+    async fn remove_file(&self, path: &str) -> Result<()> {
+        self.remove(path, false).await
+    }
+    async fn remove_dir(&self, path: &str) -> Result<()> {
+        self.remove(path, true).await
+    }
+    async fn rename(&self, from: &str, to: &str) -> Result<()> {
+        let (layer, a, b, from, to, observer) = (
+            self.layer.clone(),
+            from.to_owned(),
+            to.to_owned(),
+            self.path(from),
+            self.path(to),
+            self.observer.clone(),
+        );
+        self.process
+            .operation(async move {
+                let order = observer.order(&[(&from, true), (&to, true)]).await;
+                layer.rename(&a, &b).await?;
+                if from != to {
+                    observer.moved(&from, Some(&to));
+                }
+                drop(order);
+                (observer.changed)(&from);
+                (observer.changed)(&to);
+                Ok(())
+            })
+            .await
+    }
+}
+impl Observed {
+    async fn remove(&self, path: &str, directory: bool) -> Result<()> {
+        let (layer, local, path, observer) = (
+            self.layer.clone(),
+            path.to_owned(),
+            self.path(path),
+            self.observer.clone(),
+        );
+        self.process
+            .operation(async move {
+                let order = observer.order(&[(&path, true)]).await;
+                if directory {
+                    layer.remove_dir(&local).await?;
+                } else {
+                    layer.remove_file(&local).await?;
+                }
+                observer.moved(&path, None);
+                drop(order);
+                (observer.changed)(&path);
+                Ok(())
+            })
+            .await
+    }
+}
+#[async_trait]
+impl File for Handle {
+    async fn stat(&self) -> Result<Stat> {
+        self.file.stat().await
+    }
+    async fn read(&self, offset: u64, length: usize) -> Result<Vec<u8>> {
+        self.file.read(offset, length).await
+    }
+    async fn write(&self, offset: u64, bytes: Vec<u8>) -> Result<usize> {
+        let f = self.file.clone();
+        self.mutate(async move { f.write(offset, bytes).await })
+            .await
+    }
+    async fn append(&self, bytes: Vec<u8>) -> Result<usize> {
+        let f = self.file.clone();
+        self.mutate(async move { f.append(bytes).await }).await
+    }
+    async fn set_size(&self, size: u64) -> Result<()> {
+        let f = self.file.clone();
+        self.mutate(async move { f.set_size(size).await }).await
+    }
+    async fn sync(&self) -> Result<()> {
+        self.file.sync().await
+    }
+    async fn set_times(
+        &self,
+        accessed: Option<SystemTime>,
+        modified: Option<SystemTime>,
+    ) -> Result<()> {
+        let f = self.file.clone();
+        self.mutate(async move { f.set_times(accessed, modified).await })
+            .await
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use tokio::sync::mpsc;
+    #[test]
+    fn component_matching() {
+        assert!(under("/mailbox/a", "/mailbox"));
+        assert!(under("/mailbox", "/mailbox"));
+        assert!(under("/anything", "/"));
+        assert!(!under("/mailboxes/a", "/mailbox"));
+    }
+    /// Every visible mutation is delivered after the layer call, including rebased
+    /// open handles in another process's namespace. Removed handles stay usable.
+    #[tokio::test]
+    async fn handles_follow_rename_and_unlink() {
+        for host in [false, true] {
+            let temp = tempfile::tempdir().unwrap();
+            let process = Process::default();
+            let layer: Arc<dyn Layer> = if host {
+                Arc::new(Host::new(temp.path(), process.clone()).unwrap())
+            } else {
+                Arc::new(Memory::default())
+            };
+            let (tx, mut rx) = mpsc::unbounded_channel();
+            let observer = Observer::new(move |p| {
+                tx.send(p.to_owned()).unwrap();
+            });
+            let a = observer.layer(layer.clone(), "/run", process.clone());
+            let b = observer.layer(layer, "/run", process);
+            a.mkdir("/dir").await.unwrap();
+            assert_eq!(rx.recv().await.unwrap(), "/run/dir");
+            let f = a
+                .open(
+                    "/dir/a",
+                    Open {
+                        create: true,
+                        write: true,
+                        ..Open::default()
+                    },
+                )
+                .await
+                .unwrap();
+            assert_eq!(rx.recv().await.unwrap(), "/run/dir/a");
+            b.rename("/dir", "/moved").await.unwrap();
+            assert_eq!(rx.recv().await.unwrap(), "/run/dir");
+            assert_eq!(rx.recv().await.unwrap(), "/run/moved");
+            f.write(0, b"new".to_vec()).await.unwrap();
+            assert_eq!(rx.recv().await.unwrap(), "/run/moved/a");
+            assert_eq!(
+                b.open("/moved/a", Open::default())
+                    .await
+                    .unwrap()
+                    .read(0, 3)
+                    .await
+                    .unwrap(),
+                b"new"
+            );
+            let old = b
+                .open(
+                    "/old",
+                    Open {
+                        create: true,
+                        write: true,
+                        ..Open::default()
+                    },
+                )
+                .await
+                .unwrap();
+            rx.recv().await.unwrap();
+            b.rename("/moved/a", "/old").await.unwrap();
+            rx.recv().await.unwrap();
+            rx.recv().await.unwrap();
+            old.write(0, b"invisible".to_vec()).await.unwrap();
+            assert!(rx.try_recv().is_err());
+            f.append(b"!".to_vec()).await.unwrap();
+            assert_eq!(rx.recv().await.unwrap(), "/run/old");
+            b.remove_file("/old").await.unwrap();
+            assert_eq!(rx.recv().await.unwrap(), "/run/old");
+            f.set_size(0).await.unwrap();
+            assert!(rx.try_recv().is_err());
+            assert!(b.remove_file("/missing").await.is_err());
+            assert!(rx.try_recv().is_err());
+        }
+    }
+    /// A pending exclusive namespace change must not block a sibling's opens,
+    /// even though Tokio's RwLock prefers queued writers over new readers.
+    #[tokio::test]
+    async fn namespace_order_is_scoped_and_shared() {
+        async fn ready<T>(future: impl Future<Output = T>) -> T {
+            let mut future = std::pin::pin!(future);
+            std::future::poll_fn(|cx| match future.as_mut().poll(cx) {
+                std::task::Poll::Ready(value) => std::task::Poll::Ready(value),
+                std::task::Poll::Pending => panic!("unrelated namespace operation blocked"),
+            })
+            .await
+        }
+        let observer = Observer::new(|_| {});
+        let open = observer.order(&[("/run/slow/file", false)]).await;
+        let concurrent_open = ready(observer.order(&[("/run/slow/file", false)])).await;
+        let mut rename =
+            std::pin::pin!(observer.order(&[("/run/slow", true), ("/run/moved", true)]));
+        std::future::poll_fn(|cx| {
+            assert!(rename.as_mut().poll(cx).is_pending());
+            std::task::Poll::Ready(())
+        })
+        .await;
+        // Same mounted layer, different subtree; also a different mounted layer.
+        let sibling_open = ready(observer.order(&[("/run/sibling/file", false)])).await;
+        let host_open = ready(observer.order(&[("/state/file", false)])).await;
+        drop(sibling_open);
+        let sibling_rename =
+            ready(observer.order(&[("/run/sibling", true), ("/run/other", true)])).await;
+        drop((sibling_rename, host_open, concurrent_open, open));
+        let renamed = ready(rename).await;
+        // A descendant open waits for its ancestor's rename until rebasing is done.
+        let mut descendant = std::pin::pin!(observer.order(&[("/run/moved/file", false)]));
+        std::future::poll_fn(|cx| {
+            assert!(descendant.as_mut().poll(cx).is_pending());
+            std::task::Poll::Ready(())
+        })
+        .await;
+        drop(renamed);
+        drop(ready(descendant).await);
+        drop(
+            observer
+                .order(&[("/run/moved/file", true), ("/run/moved", true)])
+                .await,
+        );
+        drop(
+            observer
+                .order(&[("/run/moved", true), ("/run/moved", true)])
+                .await,
+        );
+        // Dead weak entries are pruned, rather than retaining every path ever opened.
+        observer.order(&[("/", false)]).await;
+        assert_eq!(observer.namespace.lock().unwrap().len(), 1);
+    }
+
+    /// Pause an open after obtaining its real identity, before registering its
+    /// observed handle, like a slow host syscall or /self/log query.
+    struct SlowOpen {
+        memory: Arc<Memory>,
+        gate: Mutex<
+            Option<(
+                tokio::sync::oneshot::Sender<()>,
+                tokio::sync::oneshot::Receiver<()>,
+            )>,
+        >,
+    }
+    #[async_trait]
+    impl Layer for SlowOpen {
+        async fn stat(&self, p: &str, follow: bool) -> Result<Stat> {
+            self.memory.stat(p, follow).await
+        }
+        async fn list(&self, p: &str) -> Result<Vec<Entry>> {
+            self.memory.list(p).await
+        }
+        async fn open(&self, p: &str, options: Open) -> Result<Arc<dyn File>> {
+            let f = self.memory.open(p, options).await?;
+            let gate = self.gate.lock().unwrap().take();
+            if let Some((started, release)) = gate {
+                started.send(()).unwrap();
+                release.await.unwrap();
+            }
+            Ok(f)
+        }
+        async fn mkdir(&self, p: &str) -> Result<()> {
+            self.memory.mkdir(p).await
+        }
+        async fn remove_file(&self, p: &str) -> Result<()> {
+            self.memory.remove_file(p).await
+        }
+        async fn remove_dir(&self, p: &str) -> Result<()> {
+            self.memory.remove_dir(p).await
+        }
+        async fn rename(&self, a: &str, b: &str) -> Result<()> {
+            self.memory.rename(a, b).await
+        }
+    }
+    #[tokio::test]
+    async fn blocked_io_does_not_stall_other_processes() {
+        let memory = Arc::new(Memory::default());
+        memory.mkdir("/slow").await.unwrap();
+        let options = Open {
+            create: true,
+            write: true,
+            ..Open::default()
+        };
+        let raw = memory.open("/slow/file", options).await.unwrap();
+        let (started, opening) = tokio::sync::oneshot::channel();
+        let (release, released) = tokio::sync::oneshot::channel();
+        let (tx, mut rx) = mpsc::unbounded_channel();
+        let observer = Observer::new(move |p| {
+            tx.send(p.to_owned()).unwrap();
+        });
+        let slow = observer.layer(
+            Arc::new(SlowOpen {
+                memory: memory.clone(),
+                gate: Mutex::new(Some((started, released))),
+            }),
+            "/run",
+            Process::default(),
+        );
+        let other = observer.layer(memory, "/run", Process::default());
+        let opening_task = tokio::spawn(async move {
+            slow.open(
+                "/slow/file",
+                Open {
+                    write: true,
+                    ..Open::default()
+                },
+            )
+            .await
+        });
+        opening.await.unwrap();
+        let renamer = other.clone();
+        let renamed = tokio::spawn(async move { renamer.rename("/slow", "/moved").await });
+        // These must complete while the first process's open is still blocked,
+        // and a conflicting rename is waiting to rebase its registered handle.
+        other
+            .open(
+                "/",
+                Open {
+                    directory: true,
+                    ..Open::default()
+                },
+            )
+            .await
+            .unwrap();
+        let f = other.open("/other", options).await.unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/other");
+        f.write(0, b"visible".to_vec()).await.unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/other");
+        other.mkdir("/sibling").await.unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/sibling");
+        other.rename("/other", "/sibling/file").await.unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/other");
+        assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
+        other.remove_file("/sibling/file").await.unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/sibling/file");
+        assert!(!renamed.is_finished());
+        release.send(()).unwrap();
+        let opened = opening_task.await.unwrap().unwrap();
+        renamed.await.unwrap().unwrap();
+        rx.recv().await.unwrap();
+        rx.recv().await.unwrap();
+        opened.set_size(0).await.unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/moved/file");
+
+        // No namespace guard surrounds descriptor IO: a rename completes while
+        // this write is held in flight. Its notification uses the rebased path.
+        let handle = Handle {
+            file: raw.clone(),
+            path: observer.path("/run/moved/file".into()),
+            observer,
+            process: Process::default(),
+        };
+        let (started, writing) = tokio::sync::oneshot::channel();
+        let (release, released) = tokio::sync::oneshot::channel();
+        let write = tokio::spawn(async move {
+            handle
+                .mutate(async move {
+                    started.send(()).unwrap();
+                    released.await.unwrap();
+                    raw.write(0, b"after rename".to_vec()).await
+                })
+                .await
+        });
+        writing.await.unwrap();
+        other.rename("/moved", "/again").await.unwrap();
+        rx.recv().await.unwrap();
+        rx.recv().await.unwrap();
+        other.open("/again/file", Open::default()).await.unwrap();
+        release.send(()).unwrap();
+        write.await.unwrap().unwrap();
+        assert_eq!(rx.recv().await.unwrap(), "/run/again/file");
+    }
+
+    /// Killing a waiting writer does not discard observation of a completed mutation;
+    /// the process's idle barrier includes the callback, not only the system call.
+    #[tokio::test]
+    async fn cancellation_keeps_mutation_and_observation() {
+        let (tx, mut rx) = mpsc::unbounded_channel();
+        let observer = Observer::new(move |p| {
+            tx.send(p.to_owned()).unwrap();
+        });
+        let process = Process::default();
+        let layer = observer.layer(Arc::new(Memory::default()), "/", process.clone());
+        let order = observer.order(&[("/published", true)]).await;
+        let (started, ready) = tokio::sync::oneshot::channel();
+        let p = process.clone();
+        let task = tokio::spawn(async move {
+            // Signal from inside the tracked operation, not before its guard exists.
+            p.operation(async move {
+                started.send(()).unwrap();
+                layer
+                    .open(
+                        "/published",
+                        Open {
+                            create: true,
+                            write: true,
+                            ..Open::default()
+                        },
+                    )
+                    .await
+            })
+            .await
+        });
+        ready.await.unwrap();
+        task.abort();
+        let _ = task.await;
+        drop(order);
+        process.idle().await;
+        assert_eq!(rx.try_recv().unwrap(), "/published");
+    }
+}

vfs/src/process.rs

@@ -36,7 +36,22 @@ impl Process {
         .await
         .map_err(std::io::Error::other)?
     }
-    /// Wait for all outstanding system calls.
+    /// Complete an asynchronous mutation and its observation even if the guest is cancelled.
+    /// Its guard participates in idle(), so no mutation outlives process cleanup.
+    pub async fn operation<T: Send + 'static>(
+        &self,
+        operation: impl std::future::Future<Output = std::io::Result<T>> + Send + 'static,
+    ) -> std::io::Result<T> {
+        self.0.count.fetch_add(1, Ordering::SeqCst);
+        let guard = Guard(self.0.clone());
+        tokio::spawn(async move {
+            let _guard = guard;
+            operation.await
+        })
+        .await
+        .map_err(std::io::Error::other)?
+    }
+    /// Wait for all outstanding system calls and observed mutations.
     /// First abort the guest task. Then await its JoinHandle, or drop its store.
     /// Only then await idle before releasing locks. Abort alone does not stop a running poll.
     /// This ordering also prevents later memory writes, which need no blocking guard.