host: add supervisor-gated clean core.restart RPC
Return success before scheduling the existing shutdown path, coalesce pending requests, and mark #loop invocations with SAUSAGE_LOOP=1. Keep core.exit compatibility and document explicit per-call approval for all agents. Card: SAU-134
This commit is contained in:
@@ -61,6 +61,19 @@ a plugin, or a change that the agent makes to itself.
|
|||||||
|
|
||||||
## Small choices
|
## Small choices
|
||||||
|
|
||||||
|
- **Restart is a clean stop supervised by `#loop`.** `core.restart` requires
|
||||||
|
the host startup marker `SAUSAGE_LOOP=1`, which the loop exports for its
|
||||||
|
invocation. Without a supervisor, restarting would only stop the world.
|
||||||
|
A single pending request answers first, allows 250 ms for delivery, then
|
||||||
|
signals the existing shutdown path; it does not use `process::exit` or
|
||||||
|
bypass detached-process cleanup, lock release, or journal draining. The
|
||||||
|
executable returns dedicated restart code **75** after cleanup. `#loop`
|
||||||
|
refreshes and starts again on 75 (and other nonzero exits); 0 stops it.
|
||||||
|
`core.exit` stays available for compatibility, but new callers use the
|
||||||
|
gated method. **HARD RULE: the mediator and all agents NEVER call
|
||||||
|
`core.restart` without the person's explicit approval each time.** This
|
||||||
|
approval rule also covers legacy `core.exit`; it must not bypass the gate.
|
||||||
|
|
||||||
- **YAML for configuration, JSON for data.** A person edits configuration, so
|
- **YAML for configuration, JSON for data.** A person edits configuration, so
|
||||||
it needs comments. `init` writes commented templates instead of asking
|
it needs comments. `init` writes commented templates instead of asking
|
||||||
questions, so there is one way to configure a world.
|
questions, so there is one way to configure a world.
|
||||||
|
|||||||
+31
-1
@@ -322,7 +322,37 @@ At start, the core starts each service in `/config/boot.yaml`, in order:
|
|||||||
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents |
|
| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents |
|
||||||
| `core.revert` | `plugin`, `force`? | like `core.activate` |
|
| `core.revert` | `plugin`, `force`? | like `core.activate` |
|
||||||
| `core.log` | filters (section 7) | journal entries |
|
| `core.log` | filters (section 7) | journal entries |
|
||||||
| `core.exit` | none | `{}`. Then the host stops with exit code 75. |
|
| `core.restart` | none | `{}`. With `SAUSAGE_LOOP=1`, schedule clean shutdown and exit code 75. |
|
||||||
|
| `core.exit` | none | `{}`. Legacy ungated shutdown with exit code 75; prefer `core.restart`. |
|
||||||
|
|
||||||
|
### Host restart
|
||||||
|
|
||||||
|
**HARD RULE: the mediator and all agents NEVER call `core.restart` without
|
||||||
|
the person's explicit approval each time.** The same approval rule applies
|
||||||
|
to the legacy `core.exit`; it is not a workaround for the restart gate.
|
||||||
|
|
||||||
|
`core.restart {}` is enabled only when the host starts with `SAUSAGE_LOOP=1`.
|
||||||
|
The host captures the marker at startup. An absent marker (or any value other
|
||||||
|
than `1`) returns error -32600: `restart needs the loop supervisor
|
||||||
|
(SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1`, without scheduling an exit.
|
||||||
|
This is a supervisor marker, not a permission system.
|
||||||
|
|
||||||
|
The method returns `{}` immediately and schedules the shutdown request after
|
||||||
|
250 milliseconds to let the response reach the caller before stdin closes.
|
||||||
|
Repeated requests while a restart is pending return `{}` without scheduling
|
||||||
|
another restart or extending that delay. The executable then uses the same
|
||||||
|
`Host::stop` path as a normal stop: reject new processes, stop all processes
|
||||||
|
(including detached ones) in parallel with the normal five-second grace,
|
||||||
|
wait for outstanding file operations, release locks, and drain journal writes.
|
||||||
|
Only after that does the executable return the dedicated restart exit code
|
||||||
|
**75**. The library signals this path; it never kills its embedding process.
|
||||||
|
|
||||||
|
`nix run .#loop -- <world>` sets `SAUSAGE_LOOP=1` on the host invocation and
|
||||||
|
starts the newest host again after exit code 75 (and, as before, other nonzero
|
||||||
|
exits). Exit code 0 ends the loop. If using an older loop script, the person
|
||||||
|
must set `SAUSAGE_LOOP=1` in that supervisor's environment; do not set it for
|
||||||
|
an unsupervised host. The NixEnv service invokes this flake's `#loop`, so a
|
||||||
|
new loop script supplies the marker without changes to NixEnv.
|
||||||
|
|
||||||
Everything else is a file.
|
Everything else is a file.
|
||||||
|
|
||||||
|
|||||||
@@ -45,9 +45,16 @@ nix run git+https://gitea.surma.technology/surma/sausage#loop # ./
|
|||||||
nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world
|
nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world
|
||||||
```
|
```
|
||||||
|
|
||||||
When the host crashes or a plugin calls `core.exit`, the loop fetches the
|
When the host crashes or a plugin calls `core.restart`, the loop fetches the
|
||||||
newest commit and starts it again. `quit` ends the loop. Set `SAUSAGE_FLAKE`
|
newest commit and starts it again. The loop sets `SAUSAGE_LOOP=1` to enable
|
||||||
to run another flake URL instead, for example your fork.
|
that RPC; without it, restart returns an error and leaves the host running.
|
||||||
|
Restart answers first, then cleanly stops all processes and exits with code
|
||||||
|
75. `quit` (code 0) ends the loop. Set `SAUSAGE_FLAKE` to run another flake
|
||||||
|
URL instead, for example your fork.
|
||||||
|
|
||||||
|
**The mediator and all agents NEVER call `core.restart` without the person's
|
||||||
|
explicit approval each time.** This also applies to legacy `core.exit`.
|
||||||
|
See [the protocol](PROTOCOL.md#host-restart) for the shutdown contract.
|
||||||
|
|
||||||
From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the
|
From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the
|
||||||
same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
|
same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
|
||||||
@@ -57,7 +64,7 @@ same commands are `nix run`, `nix run -- init`, `nix run -- run`, and
|
|||||||
|
|
||||||
`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host
|
`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host
|
||||||
runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls
|
runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls
|
||||||
`core.exit` (exit code 75, so the loop starts the newest version):
|
`core.restart` (requires `SAUSAGE_LOOP=1`; exit code 75 restarts the loop):
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world
|
nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world
|
||||||
|
|||||||
@@ -62,6 +62,7 @@ impl Host {
|
|||||||
}
|
}
|
||||||
"log" => Ok(self.0.journal.query(p).await),
|
"log" => Ok(self.0.journal.query(p).await),
|
||||||
"activate" | "revert" => self.activation(caller, method, p).await,
|
"activate" | "revert" => self.activation(caller, method, p).await,
|
||||||
|
"restart" => self.request_restart(),
|
||||||
"exit" => {
|
"exit" => {
|
||||||
self.0.exit.send_replace(true);
|
self.0.exit.send_replace(true);
|
||||||
Ok(json!({}))
|
Ok(json!({}))
|
||||||
|
|||||||
+6
-2
@@ -26,6 +26,8 @@ pub(crate) struct Inner {
|
|||||||
pub(crate) journal: Arc<journal::Journal>,
|
pub(crate) journal: Arc<journal::Journal>,
|
||||||
pub(crate) notifications: tokio::sync::broadcast::Sender<Value>,
|
pub(crate) notifications: tokio::sync::broadcast::Sender<Value>,
|
||||||
pub(crate) exit: watch::Sender<bool>,
|
pub(crate) exit: watch::Sender<bool>,
|
||||||
|
pub(crate) loop_supervised: bool,
|
||||||
|
pub(crate) restart_pending: std::sync::atomic::AtomicBool,
|
||||||
pub(crate) components: components::Components,
|
pub(crate) components: components::Components,
|
||||||
/// Calls hold it for reading from the choice of a version until their
|
/// Calls hold it for reading from the choice of a version until their
|
||||||
/// process is in the table. Activation holds it for writing while it
|
/// process is in the table. Activation holds it for writing while it
|
||||||
@@ -98,6 +100,8 @@ impl Host {
|
|||||||
}),
|
}),
|
||||||
notifications: tokio::sync::broadcast::channel(1024).0,
|
notifications: tokio::sync::broadcast::channel(1024).0,
|
||||||
exit: watch::channel(false).0,
|
exit: watch::channel(false).0,
|
||||||
|
loop_supervised: crate::restart::enabled(std::env::var_os("SAUSAGE_LOOP").as_deref()),
|
||||||
|
restart_pending: std::sync::atomic::AtomicBool::new(false),
|
||||||
components: components::Components::default(),
|
components: components::Components::default(),
|
||||||
admission: tokio::sync::RwLock::new(()),
|
admission: tokio::sync::RwLock::new(()),
|
||||||
shutdown: std::sync::atomic::AtomicBool::new(false),
|
shutdown: std::sync::atomic::AtomicBool::new(false),
|
||||||
@@ -143,12 +147,12 @@ impl Host {
|
|||||||
)
|
)
|
||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
/// Wait until a core.exit handler requests a restart.
|
/// Wait until core.restart (after its response grace) or core.exit requests shutdown.
|
||||||
pub async fn exit_requested(&self) {
|
pub async fn exit_requested(&self) {
|
||||||
let mut exit = self.0.exit.subscribe();
|
let mut exit = self.0.exit.subscribe();
|
||||||
let _ = exit.wait_for(|requested| *requested).await;
|
let _ = exit.wait_for(|requested| *requested).await;
|
||||||
}
|
}
|
||||||
/// Return whether a core.exit handler requested a restart.
|
/// Return whether a restart has reached the normal shutdown path.
|
||||||
pub fn restart_requested(&self) -> bool {
|
pub fn restart_requested(&self) -> bool {
|
||||||
*self.0.exit.borrow()
|
*self.0.exit.borrow()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ mod journal;
|
|||||||
mod locks;
|
mod locks;
|
||||||
mod native;
|
mod native;
|
||||||
mod process;
|
mod process;
|
||||||
|
mod restart;
|
||||||
mod routing;
|
mod routing;
|
||||||
pub mod runtime;
|
pub mod runtime;
|
||||||
mod services;
|
mod services;
|
||||||
|
|||||||
@@ -0,0 +1,119 @@
|
|||||||
|
//! Supervisor-gated restart requests; the executable still owns normal shutdown.
|
||||||
|
use crate::{Host, RpcResult, error};
|
||||||
|
use serde_json::json;
|
||||||
|
use std::{ffi::OsStr, sync::atomic::Ordering, time::Duration};
|
||||||
|
|
||||||
|
pub(crate) fn enabled(value: Option<&OsStr>) -> bool {
|
||||||
|
value == Some(OsStr::new("1"))
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Host {
|
||||||
|
pub(crate) fn request_restart(&self) -> RpcResult {
|
||||||
|
if !self.0.loop_supervised {
|
||||||
|
return Err(error(
|
||||||
|
-32600,
|
||||||
|
"restart needs the loop supervisor (SAUSAGE_LOOP not set); start with SAUSAGE_LOOP=1",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if !self.0.restart_pending.swap(true, Ordering::SeqCst) {
|
||||||
|
let host = self.clone();
|
||||||
|
tokio::spawn(async move {
|
||||||
|
// Let the RPC response reach the caller before closing any stdin,
|
||||||
|
// including that of a detached gateway carrying this response.
|
||||||
|
tokio::time::sleep(Duration::from_millis(250)).await;
|
||||||
|
host.0.exit.send_replace(true);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(json!({}))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::{runtime::Runtime, world::World};
|
||||||
|
use std::sync::Arc;
|
||||||
|
use tokio::time::timeout;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn only_explicit_loop_marker_enables_restart() {
|
||||||
|
assert!(!enabled(None));
|
||||||
|
for value in ["", "0", "true", "yes", "2"] {
|
||||||
|
assert!(!enabled(Some(OsStr::new(value))));
|
||||||
|
}
|
||||||
|
assert!(enabled(Some(OsStr::new("1"))));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn host(supervised: bool) -> (tempfile::TempDir, Host) {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
World::create(&dir.path().join("world")).unwrap();
|
||||||
|
let runtime = Runtime::new(&dir.path().join("cache")).unwrap();
|
||||||
|
let mut host = Host::test_world(&dir.path().join("world"), runtime)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
// No global environment mutation: parallel tests have independent gates.
|
||||||
|
Arc::get_mut(&mut host.0).unwrap().loop_supervised = supervised;
|
||||||
|
(dir, host)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn restart_without_supervisor_never_triggers_exit() {
|
||||||
|
timeout(Duration::from_secs(10), async {
|
||||||
|
let (_dir, host) = host(false).await;
|
||||||
|
let error = host.call("core.restart", json!({})).await.unwrap_err();
|
||||||
|
assert_eq!(error["code"], -32600);
|
||||||
|
assert!(
|
||||||
|
error["message"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("SAUSAGE_LOOP not set")
|
||||||
|
);
|
||||||
|
assert!(!host.0.restart_pending.load(Ordering::SeqCst));
|
||||||
|
assert!(
|
||||||
|
timeout(Duration::from_millis(300), host.exit_requested())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn restart_answers_before_trigger_and_duplicates_are_noops() {
|
||||||
|
timeout(Duration::from_secs(10), async {
|
||||||
|
let (_dir, host) = host(true).await;
|
||||||
|
// Same dispatch as a plugin's core.call, without killing this runner.
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.call", json!({"method":"core.restart"}))
|
||||||
|
.await
|
||||||
|
.unwrap(),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
assert!(!host.restart_requested());
|
||||||
|
assert!(host.0.restart_pending.load(Ordering::SeqCst));
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.restart", json!({})).await.unwrap(),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
timeout(Duration::from_secs(1), host.exit_requested())
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
// A duplicate after the trigger must not schedule a second task.
|
||||||
|
host.0.exit.send_replace(false);
|
||||||
|
assert_eq!(
|
||||||
|
host.call("core.restart", json!({})).await.unwrap(),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
timeout(Duration::from_millis(300), host.exit_requested())
|
||||||
|
.await
|
||||||
|
.is_err()
|
||||||
|
);
|
||||||
|
host.stop().await;
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1931,6 +1931,64 @@ async fn detached_processes_need_explicit_termination() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// PROTOCOL 9: restart answers the detached caller, then uses normal stop and status 75.
|
||||||
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
|
async fn core_restart_cleanly_stops_detached_caller() {
|
||||||
|
tokio::time::timeout(std::time::Duration::from_secs(30), async {
|
||||||
|
let w = TestWorld::new();
|
||||||
|
std::fs::write(
|
||||||
|
w.0.join("config/boot.yaml"),
|
||||||
|
br#"[{"method":"probe.restart","params":{}}]"#,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
let output = tokio::process::Command::new(env!("CARGO_BIN_EXE_sausage"))
|
||||||
|
.env("SAUSAGE_LOOP", "1")
|
||||||
|
.arg("--headless")
|
||||||
|
.arg("--cache-dir")
|
||||||
|
.arg(concat!(env!("CARGO_TARGET_TMPDIR"), "/cache"))
|
||||||
|
.arg("run")
|
||||||
|
.arg(&w.0)
|
||||||
|
.stdin(std::process::Stdio::null())
|
||||||
|
.kill_on_drop(true)
|
||||||
|
.output()
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
output.status.code(),
|
||||||
|
Some(75),
|
||||||
|
"{}",
|
||||||
|
String::from_utf8_lossy(&output.stderr)
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(w.0.join("state/restart-answer")).unwrap(),
|
||||||
|
"received"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
std::fs::read_to_string(w.0.join("state/restart-stop")).unwrap(),
|
||||||
|
"persisted after EOF"
|
||||||
|
);
|
||||||
|
let db = rusqlite::Connection::open(w.0.join("state/core/journal.sqlite")).unwrap();
|
||||||
|
let count: i64 = db
|
||||||
|
.query_row(
|
||||||
|
"SELECT count(*) FROM journal WHERE line = 'restart stop hook finished'",
|
||||||
|
[],
|
||||||
|
|row| row.get(0),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(count, 1);
|
||||||
|
let killed: i64 = db
|
||||||
|
.query_row(
|
||||||
|
"SELECT count(*) FROM journal WHERE line = 'kill'",
|
||||||
|
[],
|
||||||
|
|row| row.get(0),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(killed, 0);
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
/// PROTOCOL 9: core.exit from boot ends the host executable with status 75.
|
/// PROTOCOL 9: core.exit from boot ends the host executable with status 75.
|
||||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||||
async fn core_exit_sets_executable_status_75() {
|
async fn core_exit_sets_executable_status_75() {
|
||||||
|
|||||||
@@ -447,6 +447,21 @@ fn dispatch(request: &Value, lines: &mut Lines) -> Option<Value> {
|
|||||||
burst(request);
|
burst(request);
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
|
"restart" => {
|
||||||
|
assert_eq!(
|
||||||
|
lock(&json!({"path":"/state/restart-stop"}), lines),
|
||||||
|
json!({"held":true})
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
outgoing_call(&json!({"method":"core.restart", "params":{}}), lines),
|
||||||
|
json!({})
|
||||||
|
);
|
||||||
|
std::fs::write("/state/restart-answer", "received").unwrap();
|
||||||
|
serve(request, lines);
|
||||||
|
std::fs::write("/state/restart-stop", "persisted after EOF").unwrap();
|
||||||
|
eprintln!("restart stop hook finished");
|
||||||
|
return None;
|
||||||
|
}
|
||||||
"serve" => {
|
"serve" => {
|
||||||
serve(request, lines);
|
serve(request, lines);
|
||||||
return None;
|
return None;
|
||||||
|
|||||||
@@ -80,7 +80,7 @@
|
|||||||
# `nix run .#loop -- <world>` runs the newest `sausage run <world>`
|
# `nix run .#loop -- <world>` runs the newest `sausage run <world>`
|
||||||
# (without a folder: bare `sausage`, i.e. ./world, created if needed)
|
# (without a folder: bare `sausage`, i.e. ./world, created if needed)
|
||||||
# from the public repo, again and again: after a crash or a call of
|
# from the public repo, again and again: after a crash or a call of
|
||||||
# `core.exit` it refreshes the flake and starts the host again, so
|
# `core.restart` (exit code 75) it refreshes the flake and starts the host again, so
|
||||||
# the world updates itself. `quit` (exit code 0) ends the loop.
|
# the world updates itself. `quit` (exit code 0) ends the loop.
|
||||||
# SAUSAGE_FLAKE overrides where it fetches from (any flake URL,
|
# SAUSAGE_FLAKE overrides where it fetches from (any flake URL,
|
||||||
# e.g. a fork or a local checkout).
|
# e.g. a fork or a local checkout).
|
||||||
@@ -90,7 +90,7 @@
|
|||||||
# No folder: bare `sausage` (creates ./world first if needed).
|
# No folder: bare `sausage` (creates ./world first if needed).
|
||||||
if [ $# -gt 0 ]; then set -- run "$@"; fi
|
if [ $# -gt 0 ]; then set -- run "$@"; fi
|
||||||
while true; do
|
while true; do
|
||||||
nix run --refresh "$url#default" -- "$@"
|
SAUSAGE_LOOP=1 nix run --refresh "$url#default" -- "$@"
|
||||||
code=$?
|
code=$?
|
||||||
if [ "$code" -eq 0 ]; then exit 0; fi
|
if [ "$code" -eq 0 ]; then exit 0; fi
|
||||||
echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2
|
echo "sausage-loop: sausage exited with $code, starting the newest version in 2 s …" >&2
|
||||||
|
|||||||
Reference in New Issue
Block a user