Compare commits

...
Author SHA1 Message Date
Sausage Agent 5e701ab846 host: mount owner-written non-secret global config
ci / check (push) Successful in 10m37s
2026-10-22 13:05:03 +00:00
surma 3d3f6f35fb Merge pull request 'broker: upgrade and rollback keep a stopped actor stopped' (#3) from upgrade-keeps-stopped into main
ci / check (push) Successful in 13s
Reviewed-on: #3
2026-10-22 10:50:15 +01:00
surma f1af07506f broker: upgrade and rollback keep a stopped actor stopped
ci / check (push) Successful in 8m42s
ci / check (pull_request) Successful in 4m38s
A stopped actor gets the new version installed, keeps its mailbox and does
not run main until the next start. Running actors still restart on the new
version. The upgraded and rolled_back answers say whether the actor runs.
2026-10-22 10:11:36 +01:00
9 changed files with 214 additions and 16 deletions
+29 -4
View File
@@ -59,13 +59,15 @@ it exits. After 5 seconds, it kills the instance.
## Filesystem
Each instance sees three folders:
Each instance sees these mounts:
| Path | Access | Contents |
| --- | --- | --- |
| `/data` | read-write by its actor; readable by every actor | the actor's state |
| `/public` | read-write | the files that the actor publishes |
| `/actor` | read-only | the `actor/` folder of the world |
| `/config` | read-only | one non-secret global JSON file, owned by `shared` |
| `/global` | read-write for `shared`, read-only for all other actors | backing directory containing that same `config` file |
An actor can write only its own `/data`, but can read every actor's `/data` through `/actor/<id>/data/`. `/data` is not private: any unprotected secret file there is readable by every actor.
@@ -77,6 +79,28 @@ Files are plain POSIX files: a write is visible at once. To publish a file
so that readers never see half of it, write a temporary file and `rename` it
(the Rust SDK has `write_atomic`).
`/config` is global configuration, never credentials (in particular, never the
LLM proxy token). The host initializes it to `{}` without reading or copying
legacy actor settings. Configuration schema and typed edits belong to `shared`,
not the host. Ordinary actors cannot write either `/config` or its alias
`/global/config`: WASI mounts are read-only, with no message sender checks.
WASI preopens directories rather than individual files. The host mounts the
isolated world `global/` directory at `/` read-only and at `/global` with the
owner's write access. `shared` publishes atomically by writing a sibling such
as `/global/config.tmp`, closing it, then renaming it to `/global/config`.
New opens of `/config` see the replacement, including in already-running
actors; an already-open file retains its old inode. The root mount does not
expose the rest of the host world. Candidate tests get a scratch copy of
`global/`, so even `shared` tests cannot modify the live file.
On disk, the one JSON file is `<world>/global/config`. `<world>/config` is a
relative symlink to it, for native/world filesystem readers (including an
export of the world directory). Host-side atomic writers must replace
`<world>/global/config`, not replace the symlink. Starting an older world
adds the empty file and alias without changing actor state; an existing
regular `<world>/config` is moved into the backing directory on first use.
Send large or binary data as a path under `/actor/<id>/public/`, not inside a
message.
@@ -125,8 +149,8 @@ the request: the request's `type` in past tense (`spawned`, `upgraded`,
| `type` | Fields | Effect |
| --- | --- | --- |
| `spawn` | `actor`, `wasm`, `source`?, `built_by`? | test, then start a new actor |
| `upgrade` | `actor`, `wasm`, `source`?, `built_by`? | test, then swap the version |
| `rollback` | `actor`, `version` | go back to an older version |
| `upgrade` | `actor`, `wasm`, `source`?, `built_by`? | test, then swap the version: a running actor is stopped (normal shutdown) and starts the new version; a stopped actor gets the new version and stays stopped, with its mailbox untouched, until `start`. The answer has `running` |
| `rollback` | `actor`, `version` | go back to an older version; like `upgrade`, a stopped actor stays stopped. The answer has `running` |
| `stop`, `start` | `actor` | stop or start an actor |
| `remove` | `actor` | stop the actor (normal shutdown), then delete its whole folder `/actor/<id>/` (all versions, `data/`, `public/`, logs); answer `{"type": "removed", "actor"}`. Messages to it are then undeliverable, and the ID can be spawned again. An unknown actor or a native one (`broker`, `console`, `cargo`, `fs`) is an `error` |
| `subscribe`, `unsubscribe` | `actor` | receive (or not) what `actor` publishes |
@@ -212,7 +236,8 @@ instance. A version without test files passes.
**Swap.** The broker holds the mailbox, stops the old instance, makes the
top-level `actor.wasm` and `source/` a copy of the new version, and starts
it. A rollback does the same with the older version.
it. A stopped actor is not started: it keeps its mailbox and runs the new
version at the next `start`. A rollback does the same with the older version.
**Crashes.** The broker restarts a crashed actor after 0.5 s, then doubles
the delay up to 30 s. A new version that crashes 3 times within 30 seconds
+9 -4
View File
@@ -255,6 +255,10 @@ impl Broker {
}
InstanceConfig {
mounts: vec![
// WASI preopens directories, not files. This isolated directory
// supplies /config without exposing the rest of the world.
Mount { guest: "/".into(), host: scratch.map_or_else(|| self.world.global_dir(), |p| p.join("global")), writable: false },
Mount { guest: "/global".into(), host: scratch.map_or_else(|| self.world.global_dir(), |p| p.join("global")), writable: id == "shared" },
Mount { guest: "/actor".into(), host: self.world.actors_root(), writable: false },
Mount { guest: "/public".into(), host: own.join("public"), writable: true },
Mount { guest: "/data".into(), host: own.join("data"), writable: true },
@@ -331,12 +335,13 @@ impl Broker {
Ok((component, tests))
}
/// Make `version` the current `actor.wasm`, `source/`, and `actor.json`.
pub fn install(&self, id: &str, version: u64) -> Result<()> {
/// Make `version` the current `actor.wasm`, `source/`, and `actor.json`,
/// and record whether the actor runs.
pub fn install(&self, id: &str, version: u64, running: bool) -> Result<()> {
self.world.install_files(id, version)?;
self.update_meta(id, |m| {
m.version = Some(version);
m.running = true;
m.running = running;
})
}
@@ -360,7 +365,7 @@ impl Broker {
async fn create_version(self: &Arc<Self>, id: &str, meta: &VersionMeta, wasm: &Path, source: Option<&Path>) -> Result<Value> {
let (_, tests) = self.try_version(id, meta, wasm, source).await?;
self.install(id, meta.version)?;
self.install(id, meta.version, true)?;
self.start_supervisor(id, meta.version, true);
Ok(json!({"type": "spawned", "actor": id, "version": meta.version, "tests": tests}))
}
+1 -1
View File
@@ -97,7 +97,7 @@ impl Candidate<'_> {
let dir = self.broker.world.scratch_dir().join(format!("{}-{}-{n}", self.id, std::process::id()));
let scratch = Scratch { dir: dir.clone() };
let world = &self.broker.world;
for (from, name) in [(world.data_dir(self.id), "data"), (world.public_dir(self.id), "public")] {
for (from, name) in [(world.data_dir(self.id), "data"), (world.public_dir(self.id), "public"), (world.global_dir(), "global")] {
std::fs::create_dir_all(dir.join(name))?;
if from.is_dir() {
world::copy_tree(&from, &dir.join(name))?;
+1
View File
@@ -48,6 +48,7 @@ impl Host {
if lock.try_lock().is_err() {
bail!("another sausage process runs {}", root.display());
}
world.ensure_config()?;
let broker = Broker::new(world.clone(), runtime::Runtime::new()?)?;
world::write_atomic(&world.public_dir("broker").join("PROTOCOL.md"), PROTOCOL.as_bytes())?;
+3 -1
View File
@@ -23,7 +23,9 @@ use wasmtime_wasi_http::{WasiHttpCtx, WasiHttpCtxView, WasiHttpHooks, WasiHttpVi
use crate::broker::Broker;
use crate::vfs::{Vfs, VfsState, VfsView};
/// A host folder that the guest sees at `guest`.
/// A host folder that the guest sees at `guest`. File-like root paths such
/// as `/config` use an isolated backing directory mounted at `/`; owners
/// get a separate writable directory alias for atomic replacement.
#[derive(Clone, Debug)]
pub struct Mount {
pub guest: String,
+11 -6
View File
@@ -220,17 +220,16 @@ impl Supervisor {
/// Make `version` current. It is compiled first (unless `component` is
/// given): if that fails, nothing changes. `data/` and `public/` stay as
/// they are.
/// they are. A stopped actor stays stopped.
async fn install(&mut self, version: u64, component: Option<Component>) -> Result<()> {
let component = match component {
Some(c) => c,
None => self.broker.load_version(&self.id, version)?,
};
self.stop().await;
self.broker.install(&self.id, version)?;
self.broker.install(&self.id, version, self.running)?;
self.version = version;
self.component = Some(component);
self.running = true;
self.restarts = 0;
self.restart_at = None;
Ok(())
@@ -239,20 +238,26 @@ impl Supervisor {
async fn upgrade(&mut self, version: u64, component: Component, requester: String) -> Result<Value> {
let previous = self.version;
self.install(version, Some(component)).await?;
if !self.running {
// Installed, not started: main runs and mail is read at the next `start`.
return Ok(json!({"type": "upgraded", "actor": self.id, "version": version, "previous": previous, "running": false}));
}
self.probation = Some(Probation { previous, since: Instant::now(), crashes: 0, requester });
if let Err(e) = self.start().await {
self.rollback(previous).await?;
bail!("version {version} did not start ({e:#}); rolled back to {previous}");
}
Ok(json!({"type": "upgraded", "actor": self.id, "version": version, "previous": previous}))
Ok(json!({"type": "upgraded", "actor": self.id, "version": version, "previous": previous, "running": true}))
}
async fn rollback(&mut self, version: u64) -> Result<Value> {
let from = self.version;
self.install(version, None).await?;
self.probation = None;
self.start().await?;
Ok(json!({"type": "rolled_back", "actor": self.id, "from": from, "to": version}))
if self.running {
self.start().await?;
}
Ok(json!({"type": "rolled_back", "actor": self.id, "from": from, "to": version, "running": self.running}))
}
/// Handle a control request. Returns `true` when the supervisor ends.
+2
View File
@@ -6,6 +6,8 @@
//! actor `<id>` with a message from `fs` and waits up to `ASK_TIMEOUT` for
//! the reply. Other paths go to `wasmtime-wasi` unchanged except that an
//! actor directory listing advertises `fs/` when its source docs declare it.
//! `/config` and `/global/config` are native mount paths: they deliberately
//! use WASI mount permissions rather than actor-served write requests.
//!
//! - `fs_read {path}` -> `fs_file {text | base64}`, `fs_dir {entries: [{name, type}]}`
//! - `fs_write {path, text | base64}`, `fs_mkdir {path}`, `fs_remove {path}`,
+37
View File
@@ -4,6 +4,8 @@
//!
//! ```text
//! <world>/
//! global/config non-secret global JSON (shared owns writes)
//! config -> global/config world-facing read alias
//! actor/<id>/
//! actor.wasm the current version
//! actor.json version, running, subscriptions, tapped, watches (broker)
@@ -89,6 +91,37 @@ impl World {
Self { root: root.into() }
}
/// Isolated backing directory for the root /config mount and the
/// owner's writable /global alias. Never copy legacy actor config here.
pub fn global_dir(&self) -> PathBuf {
self.root.join("global")
}
pub fn config_path(&self) -> PathBuf {
self.global_dir().join("config")
}
/// Add the non-secret file to new and pre-existing worlds. The symlink
/// is for native world readers; actor mounts use the backing directory.
pub fn ensure_config(&self) -> Result<()> {
std::fs::create_dir_all(self.global_dir())?;
let alias = self.root.join("config");
if !self.config_path().exists() {
if alias.is_file() && !alias.is_symlink() {
std::fs::rename(&alias, self.config_path())?;
} else {
write_atomic(&self.config_path(), b"{}\n")?;
}
}
match std::fs::read_link(&alias) {
Ok(target) if target == Path::new("global/config") => {}
Ok(_) => bail!("{} must point to global/config", alias.display()),
Err(_) if alias.try_exists()? => bail!("{} already exists", alias.display()),
Err(_) => std::os::unix::fs::symlink("global/config", alias)?,
}
Ok(())
}
pub fn actors_root(&self) -> PathBuf {
self.root.join("actor")
}
@@ -131,6 +164,7 @@ impl World {
for id in NATIVE {
world.ensure_actor_dirs(id)?;
}
world.ensure_config()?;
Ok(world)
}
@@ -209,6 +243,9 @@ impl World {
let prefixed = |prefix: &str| {
path.strip_prefix(prefix).filter(|rest| rest.is_empty() || rest.starts_with('/'))
};
if path == "/config" || path == "/global/config" {
return Ok(self.config_path());
}
if let Some(rest) = prefixed("/actor") {
return under(self.actors_root(), rest);
}
+121
View File
@@ -47,6 +47,83 @@ async fn receive_shutdown_messages(
.expect("shutdown output or lifecycle reply did not arrive");
}
#[tokio::test]
async fn global_config_is_readable_owner_written_and_atomically_replaced() {
let world = create_world("global-config");
let (host, mut mailbox) = start(&world).await;
let wasm = toolbox_wasm().clone();
for actor in ["shared", "reader"] {
let source = passing_source(&world, actor);
spawn_type(&host, &mut mailbox, actor, &wasm, &source).await;
}
let initial = ask_type(&host, &mut mailbox, "reader", json!({"type": "read", "path": "/config"}), "content").await;
assert_eq!(initial.body["text"], "{}\n");
for path in ["/config", "/global/config", "/global/config.tmp", "/actor/shared/data/../../../../global/config"] {
ask_type(&host, &mut mailbox, "reader", json!({"type": "write", "path": path, "text": "not allowed"}), "error").await;
}
ask_type(&host, &mut mailbox, "reader", json!({"type": "rename", "path": "/config", "to": "/global/stolen"}), "error").await;
// The root view stays read-only even for the owner; publication uses its
// writable directory alias, with a sibling temporary file and rename.
ask_type(&host, &mut mailbox, "shared", json!({"type": "write", "path": "/config", "text": "not allowed"}), "error").await;
let next = "{\"llm\":{\"thinking\":\"medium\"}}\n";
ask_type(&host, &mut mailbox, "shared", json!({"type": "write", "path": "/global/config.tmp", "text": next}), "ok").await;
let before = ask_type(&host, &mut mailbox, "reader", json!({"type": "read", "path": "/config"}), "content").await;
assert_eq!(before.body["text"], "{}\n");
// A native/world reader holding the old inode sees the complete old file.
let mut old = std::fs::File::open(world.root.join("config")).unwrap();
ask_type(&host, &mut mailbox, "shared", json!({"type": "rename", "path": "/global/config.tmp", "to": "/global/config"}), "ok").await;
let after = ask_type(&host, &mut mailbox, "reader", json!({"type": "read", "path": "/config"}), "content").await;
assert_eq!(after.body["text"], next);
assert_eq!(std::fs::read_to_string(world.root.join("config")).unwrap(), next);
assert_eq!(std::fs::read_to_string(world.resolve("reader", "/config").unwrap()).unwrap(), next);
let mut previous = String::new();
std::io::Read::read_to_string(&mut old, &mut previous).unwrap();
assert_eq!(previous, "{}\n");
assert!(!world.global_dir().join("config.tmp").exists());
// The mount must not give shared writable access to any other world data.
ask_type(&host, &mut mailbox, "shared", json!({"type": "write", "path": "/global/../actor/reader/data/escape", "text": "no"}), "error").await;
host.stop().await;
let (host, mut mailbox) = start(&world).await;
let persisted = ask_type(&host, &mut mailbox, "reader", json!({"type": "read", "path": "/config"}), "content").await;
assert_eq!(persisted.body["text"], next);
host.stop().await;
}
#[tokio::test]
async fn global_config_candidates_cannot_change_the_live_file() {
let world = create_world("candidate-global-config");
let (host, mut mailbox) = start(&world).await;
let source = source_with_test(&world, "shared", json!({"steps": [
{"send": {"type": "write", "path": "/global/next", "text": "{\"test\":true}"}, "expect": {"type": "ok"}},
{"send": {"type": "rename", "path": "/global/next", "to": "/global/config"}, "expect": {"type": "ok"}},
{"send": {"type": "read", "path": "/config"}, "expect": {"type": "content", "text": "{\"test\":true}"}}
]}));
spawn_type(&host, &mut mailbox, "shared", toolbox_wasm(), &source).await;
assert_eq!(std::fs::read_to_string(world.config_path()).unwrap(), "{}\n");
host.stop().await;
}
#[tokio::test]
async fn global_config_is_added_to_existing_worlds_without_copying_actor_settings() {
let world = create_world("old-world-config");
std::fs::remove_file(world.root.join("config")).unwrap();
std::fs::remove_dir_all(world.global_dir()).unwrap();
world.ensure_actor_dirs("shared").unwrap();
// Synthetic marker only; no actual credentials are read in this test.
std::fs::write(world.data_dir("shared").join("config.json"), "legacy-private-marker").unwrap();
let (host, _) = start(&world).await;
assert_eq!(std::fs::read_to_string(world.root.join("config")).unwrap(), "{}\n");
assert_eq!(std::fs::read_to_string(world.data_dir("shared").join("config.json")).unwrap(), "legacy-private-marker");
host.stop().await;
// Preserve an operator-provided global config when adding the mounts.
std::fs::remove_file(world.root.join("config")).unwrap();
std::fs::remove_dir_all(world.global_dir()).unwrap();
std::fs::write(world.root.join("config"), "{\"operator\":true}").unwrap();
let (host, _) = start(&world).await;
assert_eq!(std::fs::read_to_string(world.root.join("config")).unwrap(), "{\"operator\":true}");
host.stop().await;
}
#[tokio::test]
async fn routes_replies_publishes_and_reports_undeliverable() {
let world = create_world("routing");
@@ -506,6 +583,50 @@ async fn rolls_back_a_crash_loop_to_the_previous_version_without_restoring_data(
host.stop().await;
}
#[tokio::test]
async fn upgrade_and_rollback_keep_a_stopped_actor_stopped() {
let world = create_world("upgrade-stopped");
let wasm = toolbox_wasm().clone();
let (host, mut mailbox) = start(&world).await;
let source = passing_source(&world, "source");
spawn_type(&host, &mut mailbox, "sleeper", &wasm, &source).await;
let stopped = request(&host, &mut mailbox, json!({"type": "stop", "actor": "sleeper"})).await;
assert_eq!(stopped.body["type"], "stopped");
let upgraded = upgrade_actor(&host, &mut mailbox, "sleeper", &wasm, &source).await;
assert_eq!(upgraded.body["type"], "upgraded", "{upgraded:?}");
assert_eq!(upgraded.body["version"], 2);
assert_eq!(upgraded.body["running"], false);
let meta = world.read_meta("sleeper");
assert_eq!((meta.version, meta.running), (Some(2), false));
// Mail waits in the mailbox: the stopped actor reads nothing.
let waiting = host.broker.send("console", Some("sleeper"), None, None, json!({"type": "version"}));
assert!(
tokio::time::timeout(Duration::from_millis(500), receive_reply(&mut mailbox, &waiting, Duration::from_secs(10)))
.await
.is_err(),
"a stopped actor must not answer"
);
let rolled_back = request(&host, &mut mailbox, json!({"type": "rollback", "actor": "sleeper", "version": 1})).await;
assert_eq!(rolled_back.body["type"], "rolled_back", "{rolled_back:?}");
assert_eq!(rolled_back.body["running"], false);
assert!(!world.read_meta("sleeper").running);
// `start` runs the installed version and delivers the waiting message.
let started = request(&host, &mut mailbox, json!({"type": "start", "actor": "sleeper"})).await;
assert_eq!(started.body["type"], "started");
let answer = receive_reply(&mut mailbox, &waiting, Duration::from_secs(10)).await;
assert_eq!(answer.body["version"], "1");
// A running actor still restarts on the new version.
let running = upgrade_actor(&host, &mut mailbox, "sleeper", &wasm, &source).await;
assert_eq!(running.body["running"], true);
assert_eq!(ask(&host, &mut mailbox, "sleeper", json!({"type": "version"})).await.body["version"], "3");
host.stop().await;
}
#[tokio::test]
async fn explicit_rollback_keeps_current_data() {
let world = create_world("explicit-rollback");