host: add supervisor-gated clean core.restart RPC

Return success before scheduling the existing shutdown path, coalesce pending
requests, and mark #loop invocations with SAUSAGE_LOOP=1. Keep core.exit
compatibility and document explicit per-call approval for all agents.

Card: SAU-134
This commit is contained in:
SAU-134 implementer
2026-10-26 09:16:10 +00:00
parent 895be183c9
commit 8c46d9b9d0
10 changed files with 257 additions and 9 deletions
+13
View File
@@ -61,6 +61,19 @@ a plugin, or a change that the agent makes to itself.
## Small choices ## Small choices
- **Restart is a clean stop supervised by `#loop`.** `core.restart` requires
the host startup marker `SAUSAGE_LOOP=1`, which the loop exports for its
invocation. Without a supervisor, restarting would only stop the world.
A single pending request answers first, allows 250 ms for delivery, then
signals the existing shutdown path; it does not use `process::exit` or
bypass detached-process cleanup, lock release, or journal draining. The
executable returns dedicated restart code **75** after cleanup. `#loop`
refreshes and starts again on 75 (and other nonzero exits); 0 stops it.
`core.exit` stays available for compatibility, but new callers use the
gated method. **HARD RULE: the mediator and all agents NEVER call
`core.restart` without the person's explicit approval each time.** This
approval rule also covers legacy `core.exit`; it must not bypass the gate.
- **YAML for configuration, JSON for data.** A person edits configuration, so - **YAML for configuration, JSON for data.** A person edits configuration, so
it needs comments. `init` writes commented templates instead of asking it needs comments. `init` writes commented templates instead of asking
questions, so there is one way to configure a world. questions, so there is one way to configure a world.
+31 -1
View File
@@ -322,7 +322,37 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents | | `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents |
| `core.revert` | `plugin`, `force`? | like `core.activate` | | `core.revert` | `plugin`, `force`? | like `core.activate` |
| `core.log` | filters (section 7) | journal entries | | `core.log` | filters (section 7) | journal entries |
| `core.exit` | none | `{}`. Then the host stops with exit code 75. | | `core.restart` | none | `{}`. With `SAUSAGE_LOOP=1`, schedule clean shutdown and exit code 75. |
| `core.exit` | none | `{}`. Legacy ungated shutdown with exit code 75; prefer `core.restart`. |
### Host restart
**HARD RULE: the mediator and all agents NEVER call `core.restart` without
the person's explicit approval each time.** The same approval rule applies
to the legacy `core.exit`; it is not a workaround for the restart gate.
`core.restart {}` is enabled only when the host starts with `SAUSAGE_LOOP=1`.
The host captures the marker at startup. An absent marker (or any value other
than `1`) returns error -32600: `restart needs the loop supervisor
(SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1`, without scheduling an exit.
This is a supervisor marker, not a permission system.
The method returns `{}` immediately and schedules the shutdown request after
250 milliseconds to let the response reach the caller before stdin closes.
Repeated requests while a restart is pending return `{}` without scheduling
another restart or extending that delay. The executable then uses the same
`Host::stop` path as a normal stop: reject new processes, stop all processes
(including detached ones) in parallel with the normal five-second grace,
wait for outstanding file operations, release locks, and drain journal writes.
Only after that does the executable return the dedicated restart exit code
**75**. The library signals this path; it never kills its embedding process.
`nix run .#loop -- <world>` sets `SAUSAGE_LOOP=1` on the host invocation and
starts the newest host again after exit code 75 (and, as before, other nonzero
exits). Exit code 0 ends the loop. If using an older loop script, the person
must set `SAUSAGE_LOOP=1` in that supervisor's environment; do not set it for
an unsupervised host. The NixEnv service invokes this flake's `#loop`, so a
new loop script supplies the marker without changes to NixEnv.
Everything else is a file. Everything else is a file.
+11 -4
View File
@@ -45,9 +45,16 @@ nix run git+https://gitea.surma.technology/surma/sausage#loop # ./
nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world
``` ```
When the host crashes or a plugin calls `core.exit`, the loop fetches the When the host crashes or a plugin calls `core.restart`, the loop fetches the
newest commit and starts it again. `quit` ends the loop. Set `SAUSAGE_FLAKE` newest commit and starts it again. The loop sets `SAUSAGE_LOOP=1` to enable
to run another flake URL instead, for example your fork. that RPC; without it, restart returns an error and leaves the host running.
Restart answers first, then cleanly stops all processes and exits with code
75. `quit` (code 0) ends the loop. Set `SAUSAGE_FLAKE` to run another flake
URL instead, for example your fork.
**The mediator and all agents NEVER call `core.restart` without the person's
explicit approval each time.** This also applies to legacy `core.exit`.
See [the protocol](PROTOCOL.md#host-restart) for the shutdown contract.
From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the
same commands are `nix run`, `nix run -- init`, `nix run -- run`, and same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
@@ -57,7 +64,7 @@ same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host `--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host
runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls
`core.exit` (exit code 75, so the loop starts the newest version): `core.restart` (requires `SAUSAGE_LOOP=1`; exit code 75 restarts the loop):
```sh ```sh
nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world
+1
View File
@@ -62,6 +62,7 @@ impl Host {
} }
"log" => Ok(self.0.journal.query(p).await), "log" => Ok(self.0.journal.query(p).await),
"activate" | "revert" => self.activation(caller, method, p).await, "activate" | "revert" => self.activation(caller, method, p).await,
"restart" => self.request_restart(),
"exit" => { "exit" => {
self.0.exit.send_replace(true); self.0.exit.send_replace(true);
Ok(json!({})) Ok(json!({}))
+6 -2
View File
@@ -26,6 +26,8 @@ pub(crate) struct Inner {
pub(crate) journal: Arc<journal::Journal>, pub(crate) journal: Arc<journal::Journal>,
pub(crate) notifications: tokio::sync::broadcast::Sender<Value>, pub(crate) notifications: tokio::sync::broadcast::Sender<Value>,
pub(crate) exit: watch::Sender<bool>, pub(crate) exit: watch::Sender<bool>,
pub(crate) loop_supervised: bool,
pub(crate) restart_pending: std::sync::atomic::AtomicBool,
pub(crate) components: components::Components, pub(crate) components: components::Components,
/// Calls hold it for reading from the choice of a version until their /// Calls hold it for reading from the choice of a version until their
/// process is in the table. Activation holds it for writing while it /// process is in the table. Activation holds it for writing while it
@@ -98,6 +100,8 @@ impl Host {
}), }),
notifications: tokio::sync::broadcast::channel(1024).0, notifications: tokio::sync::broadcast::channel(1024).0,
exit: watch::channel(false).0, exit: watch::channel(false).0,
loop_supervised: crate::restart::enabled(std::env::var_os("SAUSAGE_LOOP").as_deref()),
restart_pending: std::sync::atomic::AtomicBool::new(false),
components: components::Components::default(), components: components::Components::default(),
admission: tokio::sync::RwLock::new(()), admission: tokio::sync::RwLock::new(()),
shutdown: std::sync::atomic::AtomicBool::new(false), shutdown: std::sync::atomic::AtomicBool::new(false),
@@ -143,12 +147,12 @@ impl Host {
) )
.await .await
} }
/// Wait until a core.exit handler requests a restart. /// Wait until core.restart (after its response grace) or core.exit requests shutdown.
pub async fn exit_requested(&self) { pub async fn exit_requested(&self) {
let mut exit = self.0.exit.subscribe(); let mut exit = self.0.exit.subscribe();
let _ = exit.wait_for(|requested| *requested).await; let _ = exit.wait_for(|requested| *requested).await;
} }
/// Return whether a core.exit handler requested a restart. /// Return whether a restart has reached the normal shutdown path.
pub fn restart_requested(&self) -> bool { pub fn restart_requested(&self) -> bool {
*self.0.exit.borrow() *self.0.exit.borrow()
} }
+1
View File
@@ -12,6 +12,7 @@ mod journal;
mod locks; mod locks;
mod native; mod native;
mod process; mod process;
mod restart;
mod routing; mod routing;
pub mod runtime; pub mod runtime;
mod services; mod services;
+119
View File
@@ -0,0 +1,119 @@
//! Supervisor-gated restart requests; the executable still owns normal shutdown.
use crate::{Host, RpcResult, error};
use serde_json::json;
use std::{ffi::OsStr, sync::atomic::Ordering, time::Duration};
pub(crate) fn enabled(value: Option<&OsStr>) -> bool {
value == Some(OsStr::new("1"))
}
impl Host {
pub(crate) fn request_restart(&self) -> RpcResult {
if !self.0.loop_supervised {
return Err(error(
-32600,
"restart needs the loop supervisor (SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1",
));
}
if !self.0.restart_pending.swap(true, Ordering::SeqCst) {
let host = self.clone();
tokio::spawn(async move {
// Let the RPC response reach the caller before closing any stdin,
// including that of a detached gateway carrying this response.
tokio::time::sleep(Duration::from_millis(250)).await;
host.0.exit.send_replace(true);
});
}
Ok(json!({}))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{runtime::Runtime, world::World};
use std::sync::Arc;
use tokio::time::timeout;
#[test]
fn only_explicit_loop_marker_enables_restart() {
assert!(!enabled(None));
for value in ["", "0", "true", "yes", "2"] {
assert!(!enabled(Some(OsStr::new(value))));
}
assert!(enabled(Some(OsStr::new("1"))));
}
async fn host(supervised: bool) -> (tempfile::TempDir, Host) {
let dir = tempfile::tempdir().unwrap();
World::create(&dir.path().join("world")).unwrap();
let runtime = Runtime::new(&dir.path().join("cache")).unwrap();
let mut host = Host::test_world(&dir.path().join("world"), runtime)
.await
.unwrap();
// No global environment mutation: parallel tests have independent gates.
Arc::get_mut(&mut host.0).unwrap().loop_supervised = supervised;
(dir, host)
}
#[tokio::test]
async fn restart_without_supervisor_never_triggers_exit() {
timeout(Duration::from_secs(10), async {
let (_dir, host) = host(false).await;
let error = host.call("core.restart", json!({})).await.unwrap_err();
assert_eq!(error["code"], -32600);
assert!(
error["message"]
.as_str()
.unwrap()
.contains("SAUSAGE_LOOP not set")
);
assert!(!host.0.restart_pending.load(Ordering::SeqCst));
assert!(
timeout(Duration::from_millis(300), host.exit_requested())
.await
.is_err()
);
host.stop().await;
})
.await
.unwrap();
}
#[tokio::test]
async fn restart_answers_before_trigger_and_duplicates_are_noops() {
timeout(Duration::from_secs(10), async {
let (_dir, host) = host(true).await;
// Same dispatch as a plugin's core.call, without killing this runner.
assert_eq!(
host.call("core.call", json!({"method":"core.restart"}))
.await
.unwrap(),
json!({})
);
assert!(!host.restart_requested());
assert!(host.0.restart_pending.load(Ordering::SeqCst));
assert_eq!(
host.call("core.restart", json!({})).await.unwrap(),
json!({})
);
timeout(Duration::from_secs(1), host.exit_requested())
.await
.unwrap();
// A duplicate after the trigger must not schedule a second task.
host.0.exit.send_replace(false);
assert_eq!(
host.call("core.restart", json!({})).await.unwrap(),
json!({})
);
assert!(
timeout(Duration::from_millis(300), host.exit_requested())
.await
.is_err()
);
host.stop().await;
})
.await
.unwrap();
}
}
+58
View File
@@ -1931,6 +1931,64 @@ async fn detached_processes_need_explicit_termination() {
} }
} }
/// PROTOCOL 9: restart answers the detached caller, then uses normal stop and status 75.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn core_restart_cleanly_stops_detached_caller() {
tokio::time::timeout(std::time::Duration::from_secs(30), async {
let w = TestWorld::new();
std::fs::write(
w.0.join("config/boot.yaml"),
br#"[{"method":"probe.restart","params":{}}]"#,
)
.unwrap();
let output = tokio::process::Command::new(env!("CARGO_BIN_EXE_sausage"))
.env("SAUSAGE_LOOP", "1")
.arg("--headless")
.arg("--cache-dir")
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
.arg("run")
.arg(&w.0)
.stdin(std::process::Stdio::null())
.kill_on_drop(true)
.output()
.await
.unwrap();
assert_eq!(
output.status.code(),
Some(75),
"{}",
String::from_utf8_lossy(&output.stderr)
);
assert_eq!(
std::fs::read_to_string(w.0.join("state/restart-answer")).unwrap(),
"received"
);
assert_eq!(
std::fs::read_to_string(w.0.join("state/restart-stop")).unwrap(),
"persisted after EOF"
);
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap();
let count: i64 = db
.query_row(
"SELECT count(*) FROM journal WHERE line = 'restart stop hook finished'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(count, 1);
let killed: i64 = db
.query_row(
"SELECT count(*) FROM journal WHERE line = 'kill'",
[],
|row| row.get(0),
)
.unwrap();
assert_eq!(killed, 0);
})
.await
.unwrap();
}
/// PROTOCOL 9: core.exit from boot ends the host executable with status 75. /// PROTOCOL 9: core.exit from boot ends the host executable with status 75.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)] #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn core_exit_sets_executable_status_75() { async fn core_exit_sets_executable_status_75() {
+15
View File
@@ -447,6 +447,21 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
burst(request); burst(request);
return None; return None;
} }
"restart" => {
assert_eq!(
lock(&json!({"path":"/state/restart-stop"}), lines),
json!({"held":true})
);
assert_eq!(
outgoing_call(&json!({"method":"core.restart", "params":{}}), lines),
json!({})
);
std::fs::write("/state/restart-answer", "received").unwrap();
serve(request, lines);
std::fs::write("/state/restart-stop", "persisted after EOF").unwrap();
eprintln!("restart stop hook finished");
return None;
}
"serve" => { "serve" => {
serve(request, lines); serve(request, lines);
return None; return None;
+2 -2
View File
@@ -80,7 +80,7 @@
# `nix run .#loop -- <world>` runs the newest `sausage run <world>` # `nix run .#loop -- <world>` runs the newest `sausage run <world>`
# (without a folder: bare `sausage`, i.e. ./world, created if needed) # (without a folder: bare `sausage`, i.e. ./world, created if needed)
# from the public repo, again and again: after a crash or a call of # from the public repo, again and again: after a crash or a call of
# `core.exit` it refreshes the flake and starts the host again, so # `core.restart` (exit code 75) it refreshes the flake and starts the host again, so
# the world updates itself. `quit` (exit code 0) ends the loop. # the world updates itself. `quit` (exit code 0) ends the loop.
# SAUSAGE_FLAKE overrides where it fetches from (any flake URL, # SAUSAGE_FLAKE overrides where it fetches from (any flake URL,
# e.g. a fork or a local checkout). # e.g. a fork or a local checkout).
@@ -90,7 +90,7 @@
# No folder: bare `sausage` (creates ./world first if needed). # No folder: bare `sausage` (creates ./world first if needed).
if [ $# -gt 0 ]; then set -- run "$@"; fi if [ $# -gt 0 ]; then set -- run "$@"; fi
while true; do while true; do
nix run --refresh "$url#default" -- "$@" SAUSAGE_LOOP=1 nix run --refresh "$url#default" -- "$@"
code=$? code=$?
if [ "$code" -eq 0 ]; then exit 0; fi if [ "$code" -eq 0 ]; then exit 0; fi
echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2 echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2