From d1be22f65305b260c206ac32de3735110d5b485b Mon Sep 17 00:00:00 2001 From: Surma Date: Thu, 22 Oct 2026 23:50:24 +0100 Subject: [PATCH] PROTOCOL.md, DECISIONS.md, README.md, AGENTS.md for plugins, processes, and JSON-RPC --- AGENTS.md | 30 +-- DECISIONS.md | 96 ++++---- PROTOCOL.md | 632 +++++++++++++++++++++++++++++++-------------------- README.md | 85 +++---- 4 files changed, 505 insertions(+), 338 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 544eb82..b7c0c2c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,25 +10,25 @@ Keep it that way. caller. No configuration that nobody asked for. 3. **Readable.** A new reader must understand a file without the others. Each module starts with a doc comment that says what it owns. -4. **Efficient** on the paths that run per message. +4. **Efficient** on the paths that run per call. 5. **Raw and minimal.** Sausage has no permissions, sandboxing beyond WASI, limits, or authentication. Do not add them to the core. Add nothing to - the host that an actor (or the agent, by changing itself) could do. + the host that a plugin (or the agent, by changing itself) could do. ## Layout -- `cli/`: the host (`sausage` binary and library). - - `lib.rs`: `Host`, which starts and stops a world. - - `broker.rs`: routing, mailboxes, the version store, broker requests. - - `supervisor.rs`: runs one WASM actor (restarts, swaps, rollbacks). - - `runtime.rs`: Wasmtime instances and what they get (mounts, network). - - `candidate.rs`: tests a new version. `cargo_actor.rs`: builds Rust. - - `world.rs`: the on-disk layout. `console.rs`: the terminal. -- `sdk/`: `sausage-actor`, the SDK for Rust actors. Never published: the - `cargo` actor points `sausage-actor = "0.1"` at its local copy. -- `agent/`: the agent actor. `sausage init` embeds and builds it. -- `examples/`: small actors. They are not workspace members. -- `PROTOCOL.md`: the contract between the host and the actors. Change it in +- `cli/`: the host (`sausage` binary and library): the core, the native + `cargo` plugin, the WASI test runner, and the console. +- `vfs/`: `sausage-vfs`, the virtual filesystem for WASI guests. It knows + nothing about Sausage. +- `interface/`: `sausage-interface`: OpenRPC documents, the compatibility + check, and the generator of client code. +- `sdk/`: `sausage-plugin`, the SDK for Rust plugins, and `sdk/macros/`, its + macros. Never published: `cargo` points `sausage-plugin = "0.1"` at its + local copy. +- `agent/`: the agent plugin. `sausage init` embeds and builds it. +- `examples/`: small plugins. They are not workspace members. +- `PROTOCOL.md`: the contract between the core and the plugins. Change it in the same commit as the behavior. `DECISIONS.md` says why; when the two disagree, `PROTOCOL.md` and the code win. @@ -36,7 +36,7 @@ Keep it that way. ```sh nix flake check -L # all tests below; CI runs this on every push -nix develop -c cargo test -p sausage -p sausage-actor -p agent +nix develop -c cargo test --workspace nix develop -c cargo build --release -p agent --target wasm32-wasip2 nix develop -c cargo clippy --workspace ``` diff --git a/DECISIONS.md b/DECISIONS.md index be3e965..56c1588 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1,55 +1,71 @@ # Decisions -Why Sausage is the way it is. `PROTOCOL.md` and the code own the details; -when this file disagrees with them, they win. +Why Sausage is the way it is. `PROTOCOL.md` and the code own the details. +When this file disagrees with them, they win. ## The shape -- **Actors are WASI commands, not components with interfaces.** stdin, - stdout, files, HTTP, and sockets are enough, and every language can do - them. -- **Messages are JSON lines through a broker, fire and forget.** Like - Erlang: no acknowledgments, no request/response. A reply is a message with - `reply_to`. Sausage is not a reliable message bus. -- **The broker accepts only `.wasm` files.** It knows no source language. - `cargo` is one compiler actor; others can add more languages. -- **The world folder is the source of truth.** Everything about an actor is - in `actor//`. Copy a folder into another world, and it runs there. -- **The agent is a normal WASM actor.** It can read and upgrade its own - source, and change how it does inference. +- **JSON-RPC, not component interfaces.** Every language can read and write + lines of JSON, and models know JSON-RPC from LSP and MCP. A call has a + result, like an LLM tool call. +- **A process per call.** WASI has no threads, so processes give + parallelism. A new version applies at the next call, and old processes + retire after a grace period. +- **Attached by default.** A process does not outlive its caller by + accident. A long-running process must ask for `detach`. +- **Services in one file.** Only the core starts and restarts them, so + upgrades and singletons need no logic in plugins. +- **Files to inspect, calls to act.** This keeps the core API small, and the + agent's file tools are enough to explore the world. +- **The core accepts only `.wasm` files.** It knows no source language. + `cargo` is one way to get a `.wasm` file. +- **The world folder is the source of truth.** Copy a plugin folder into + another world, and it works there. +- **The agent is a normal plugin.** It can read and upgrade its own source, + and change how it does inference. + +## Interfaces + +- **Generated imports and interface checks, without version numbers.** A + model that translates a description by hand makes errors. A generator does + not. The check catches the most common error in a world that changes + itself: a change to one plugin breaks another one. +- **A small schema subset.** With Draft 7 and a fixed subset, our own check + can decide every case. It never has to guess. +- **Docs are part of the interface.** Other agents work from them, so the + SDK requires a doc comment on each method, type, and field. +- **The check covers calls, not state.** Old and new versions share + `/state`. A plugin keeps its state format compatible, or migrates it in + `rpc.retired`. ## Minimal on purpose -The harness does only what it needs to keep going. Everything else can be an -actor, or a change that the agent makes to itself. +The core does only what a plugin cannot do for itself. Everything else can be +a plugin, or a change that the agent makes to itself. -- **The host is not self-modifiable; actors are.** So the host keeps only - what an actor cannot do for itself: routing, running instances, versions, - tests before an install, and the rollback of a version that crash-loops. - Without that rollback, a broken self-upgrade would end the agent. -- **No permissions, no limits, no hiding.** Any actor can message, upgrade, - or stop any actor, read every folder, and use the network. People who want rules - can build them. -- **No re-delivery and no state snapshots.** A crash loses the message, and - all versions of an actor share `data/` and `public/`. A rollback restores - code, not data. +- **No permissions, no limits, no hiding.** Any process can call, upgrade, + or stop any plugin, read and write every file, and use the network. The + checks protect against mistakes, not against plugins. Locks are advisory. +- **Locks are in the core.** A plugin cannot see that another process died, + so it cannot release that process's lock. The core can. +- **Our own VFS, in its own crate.** It works on Linux and macOS, it makes + `/proc` live, and plugin-served files need it later. Mounts now, overlays + later. +- **TCP endpoints for now.** They need no host code. VFS pipes can replace + them later behind the same SDK API. +- **Simple scheduling for now.** An epoch tick every 10 milliseconds keeps + the core responsive and makes kills reliable. - **The agent is dumb.** It has a short prompt, a few tools, an append-only history that it sends in full, and no compaction. When the context is too large, it stops sending the oldest turns, so it never gets stuck. ## Small choices -- **Plain POSIX files.** An earlier version staged every write until close. - That broke `tempfile`, git, and SQLite, which expect normal files. Atomic - publishing is the usual temporary file + `rename`. -- **Test first, then switch.** A new version is tested where it lies. Only - a version that passes goes to `versions/`, and the old one runs until - then. `versions/` keeps every version that was installed, so a rollback - can always go back. A file that fails stays where it was put. -- **Tests are JSON transcripts** next to the source. They run on a copy of - the actor's state, without network, so a test cannot send real requests. -- **`broker` and `cargo` never answer a reply,** so no two actors answer - each other forever. The agent starts a turn only for text that is not a - reply, for the same reason. -- **The SDK is not published.** `cargo` points `sausage-actor = "0.1"` at its - own copy, which it also shows in `/actor/cargo/public/sdk/`. +- **Plain POSIX files.** Programs like `tempfile`, git, and SQLite expect + normal files. Atomic writes use a temporary file and `rename`. +- **Tests run under WASI.** `cargo test` on the host cannot find failures + that occur only in WASI, for example a missing WASI function or a panic at + start. The test runner uses the same Wasmtime and the same mounts as the + core. +- **The SDK is not published.** `cargo` points `sausage-plugin = "0.1"` at + its own copy, which it also shows in `/plugins/cargo/sdk/`. diff --git a/PROTOCOL.md b/PROTOCOL.md index 1ed21c8..ac0b13a 100644 --- a/PROTOCOL.md +++ b/PROTOCOL.md @@ -1,275 +1,425 @@ # The Sausage protocol -This is the contract between the host and the actors. An actor in any -language that follows it works in Sausage. The host publishes this file at -`/actor/broker/public/PROTOCOL.md`, so every actor can read it. +This is the contract between the core and the plugins. A plugin in any +language that follows it works in a Sausage world. The core shows this file +at `/proc/core/PROTOCOL.md`. `DECISIONS.md` in the repository says why the +contract is the way it is. -## Actors +Rust plugins use the SDK `sausage-plugin`. Its source is at +`/plugins/cargo/sdk/`, and its crate docs explain each item. -An actor is a WASI 0.2 command (`wasm32-wasip2`): a program with `main`. The -host runs one instance per actor. An instance can run forever. +## 1. Terms -The instance gets: - -- **stdin:** one message per line, from the broker -- **stdout:** one message per line, to the broker -- **stderr:** free text, appended to `/actor//stderr.log` -- **environment:** `SAUSAGE_ACTOR` (its ID), `SAUSAGE_VERSION` (its version), - and `SAUSAGE_CANDIDATE=1` during a test run -- **the network:** outgoing HTTP and HTTPS through `wasi:http` (the host does - TLS), and TCP, UDP, and DNS through `wasi:sockets`, including listening on - a port of the host -- **the filesystem** described below - -## Messages - -A line on **stdin**: - -```json -{"id": "m-17", "from": "console", "reply_to": "agent/a5", "body": {"type": "text", "text": "hi"}} -``` - -A line on **stdout**: - -```json -{"to": "console", "id": "a6", "reply_to": "m-17", "body": {"type": "text", "text": "hello"}} -``` - -- `body` is required. By convention it is an object with a `"type"`. - Text for people is `{"type": "text", "text": "..."}`. -- `to` is the recipient. Without `to`, the broker publishes the message to - every subscriber of the sender. -- `id` is an optional string. If the actor sets it, the message ID is `"/"`. - Otherwise, the broker assigns `m-`. The broker always sets `from`. -- `reply_to` marks a reply: its value is the ID of the message it answers. - There is no request/response. A reply is a new message. -- **Never answer a reply,** or two actors can answer each other forever. - `broker` and `cargo` ignore messages that have a `reply_to`. -- Messages are fire and forget: delivery is at most once. Each actor has a - mailbox of 1,000 messages, which keeps messages while the actor restarts - or is upgraded. If the broker cannot deliver a message, the sender gets - `{"type": "undeliverable", "reason": "...", "to": "..."}` from `broker`. -- An actor must keep reading stdin. If the broker must stop or upgrade an - actor while a message waits for it to read stdin, it kills the actor. - -Before the broker stops an instance, it sends -`{"id": "broker/shutdown", "from": "broker", "body": {"type": "shutdown"}}` -and then closes stdin. The broker still routes what the actor writes until -it exits. After 5 seconds, it kills the instance. - -## Filesystem - -The world folder is the foundation of every instance's filesystem: - -| Path | Access | Contents | -| --- | --- | --- | -| `/` | read-write by every actor | the whole world folder | -| `/self` | read-write | alias for the actor's own folder (`public/`, `data/`, `source/`, etc.) | -| `/data` | read-write | alias for the actor's state folder | -| `/public` | read-write | alias for the actor's published files | -| `/actor` | read-write by every actor | the `actor/` folder of the world | - -Every actor can read and write every file in the world, including other actors' -`/actor//data/` and `/actor//public/`. Nothing there is private. -Creating a folder in the world makes it visible to running actors immediately: -`/config`, `/docs`, `/www`, and `/skills` need no new mounts or host restart. -New worlds include an empty `/config` for global configuration by convention; -existing worlds can simply create it. Keep secrets out of global configuration. - -Candidate instances see the same writable world root. Their `/self` alias -points to the scratch actor folder; `/public` and `/data` still point to its -corresponding children, and network access remains off; -absolute world paths are not isolated from the live world. - -`/actor//` holds everything about an actor: `actor.wasm`, `actor.json` -(installed version, running, subscriptions, tapped, watches), `source/`, `public/`, `data/`, -`stderr.log`, and `versions//`. `ls /actor` lists all actors. - -Files are plain POSIX files: a write is visible at once. To publish a file -so that readers never see half of it, write a temporary file and `rename` it -(the Rust SDK has `write_atomic`). - -Send large or binary data as a path under `/actor//public/`, not inside a -message. - -### Actor-served files: `/actor//fs/` - -Files under `/actor//fs/` are not on disk: the actor `` serves them. -Every actor can read and write them. The host turns each file operation into a -message from `fs` to `` and waits up to 10 seconds for the reply. `path` -is relative to `fs/` and starts with `/` (the folder itself is `/`). - -To make this virtual directory discoverable, an actor that serves it declares -`Actor-served files` and `fs_read` in its leading `//!` documentation in -`source/src/main.rs`. The host uses that declaration when listing -`/actor//`; it does not probe every actor (which could stall on actors that -do not handle filesystem messages). This is discovery metadata only: opening -and reading the path still requires a valid `fs_read` reply. - -| Request | Reply | +| Term | Meaning | | --- | --- | -| `{"type": "fs_read", "path"}` | `{"type": "fs_file", "text"}` (or `"base64"` instead of `"text"`), or `{"type": "fs_dir", "entries": [{"name", "type": "file" \| "dir"}]}` | -| `{"type": "fs_write", "path", "text"}` (or `"base64"` for non-UTF-8 data) | `{"type": "fs_done"}` | -| `{"type": "fs_mkdir", "path"}`, `{"type": "fs_remove", "path"}` | `{"type": "fs_done"}` | -| `{"type": "fs_rename", "path", "to"}` | `{"type": "fs_done"}` | +| world | The folder that a `sausage` host runs. It is `/` for every process. | +| core | The host. It runs processes, routes calls, serves the virtual files, and keeps the journal. | +| plugin | A folder `/plugins//` with a `plugin.wasm`. A plugin does not run by itself. | +| process | One run of a plugin version. Its ID is `-`, for example `notes-0197f3a2-7c1e-7d2b-9a41-5f0c2e8b1d34`. | +| call | A JSON-RPC request. `notes.add` calls the method `add` of the plugin `notes`. | +| interface | A set of methods. A plugin has one main interface. It can also declare endpoint interfaces. | +| endpoint | An interface that a running process serves on a TCP port. An endpoint file tells others where it is. | +| service | A call in `/config/boot.json`. The core starts it at boot and again after an upgrade of its plugin. | +| import | An interface of another plugin that a plugin uses. | +| provider, consumer | The plugin that has an interface, and a plugin that imports it. | +| journal | The central log of the world. | -Any request can be answered with `{"type": "fs_error", "errno": "no-entry"}`. -`errno` is a `wasi:filesystem` error code: `no-entry`, `exist`, -`not-directory`, `is-directory`, `not-empty`, `access`, `not-permitted`, -`invalid`, `unsupported`, `busy`, or `io` (also for anything else). +## 2. The world folder -- Opening a file sends `fs_read` once. Reads come from that copy. -- After a change, closing (or syncing) the file sends the whole file in one - `fs_write`. -- No reply within 10 seconds, or an actor that does not exist, makes the - operation fail (`io`, or `no-entry`). An actor that opens its own `fs/` - waits for itself and fails after the timeout. -- `fs` is reserved: no actor can have that ID. It also sends `fs_changed` - to watchers (see Watches). +```text +/ + plugins// + plugin.wasm the current version + plugin.prev.wasm the version before, for `core.revert` + source/ the source. `cargo.build` builds it from here. + imports/.json descriptions of the imported plugins + state// persistent state of a plugin, by convention + state/core/journal.sqlite the journal + config/ global configuration + boot.json the services + agent.json API, base URL, model, and token of the agent +``` -## The broker +- Every process can read and write all of it. There are no permissions. +- Plugin names use `a-z`, `0-9`, `-`, and `_`. The name `core` is reserved. +- The world folder must not contain `proc`, `run`, or `self`. These names are + mount points (section 4). The host does not start a world that has them. -Requests to `broker`. The answer is a message whose `reply_to` is the ID of -the request: the request's `type` in past tense (`spawned`, `upgraded`, -`rolled_back`, `stopped`, `started`, `removed`, `subscribed`, `unsubscribed`, -`tapping`, `untapped`, `watching`, `unwatched`, `exiting`), or `{"type": "error", "reason": "..."}`. +## 3. Calls and processes -| `type` | Fields | Effect | +### 3.1 The wire protocol + +- A plugin is a `wasi:cli/command` component for `wasm32-wasip2`. +- stdin and stdout carry JSON-RPC 2.0, one message per line. +- A line is a message only if it is a JSON object with `"jsonrpc": "2.0"`. + The core discards every other line on stdout. For the first discarded line + of a process, it writes one warning to the journal. +- Each line on stderr goes to the journal. Log to stderr. +- `params` is always an object. Batches are not supported. +- Write each message in one write, and write a newline before it. Then a + `print!` without a newline cannot damage a message. + +### 3.2 Method names + +- A method name has the form `.`. The core splits it at the + first `.`. +- `notes.add` starts a new process of `notes` and sends it the request `add`. + `notes.rpc.discover` sends it `rpc.discover`. +- `core.` is a method of the core (section 9). +- Method names that start with `rpc.` are reserved: `rpc.discover`, + `rpc.retired`, and `rpc.endpoint`. + +### 3.3 What a call does + +1. The core gives the call a UUIDv7 and starts the process `-` + with the current version of the plugin. +2. The core writes the request to the stdin of the process. The request ID is + the process ID, and the method has no plugin prefix. +3. The process writes the response to stdout. The core sends the response to + the caller, with the caller's request ID. +4. If the process ends before it answers, the caller gets the error -32002. + Its `data` has the exit status and the last journal lines of the process. + Before the core decides this, it reads the stdout of the process to the + end. + +- A process can write calls to stdout at any time, to the core and to + plugins. The responses come on stdin. A process can have many calls in + flight. +- The core keys each pending request by the process and the exact JSON ID. + The core drops a response with an unknown or late ID. +- A notification to `plugin.method` (without an ID) starts an attached + process. That process gets the notification without an ID, and nobody gets + a result. + +### 3.4 Detached calls + +`core.call {"method": "www.start", "params": {...}, "detach": true}` makes the +call and returns its result to the caller. The new process has no parent, so +it does not end with its caller. + +Sent as a notification (without an ID), the same `core.call` starts the +process and forgets it. + +### 3.5 Notifications to the caller + +A process can tell its caller about progress. It sends the notification +`core.notify {"name": "text", "params": {...}}`. The core gives it to the +caller as `core.notify {"call": , "name": "text", +"params": {...}}`. After the call has ended, the core drops such +notifications. + +So a process never needs to know who called it. + +### 3.6 Lifetime + +A process lives until `main` returns. A process that uses the SDK lives +while one of these runs: + +- a handler of a call, +- a background task, +- an endpoint that listens. + +A process has three states: + +1. **running** +2. **stopping**: the core has closed stdin. EOF on stdin means "stop now". + - New calls of the process fail at once, and its pending calls fail. + - A stop hook can still write files and send notifications. It cannot + make calls, because no response can arrive. + - After 5 seconds, the core kills the process. +3. **ended**: the core releases the locks of the process (section 6) and + removes its folder in `/proc`. + +Rules: + +- `core.stop {pid}` starts the stopping state. `core.kill {pid}` kills the + process at once. +- When a process ends, the core stops each of its attached children. All + children stop in parallel, so a whole tree takes 5 seconds at most. +- A detached process has no parent. Only `core.stop`, `core.kill`, an + upgrade (section 3.7), and the end of the host stop it. +- At host shutdown, the core stops every process in parallel. + +Note: on Unix, children outlive their parent. Here, an attached child ends +with its parent. Only `detach` makes a process independent. + +### 3.7 Upgrades + +A process is an instance of one plugin version. When a new version becomes +active (section 10.3): + +1. New calls get the new version at once. +2. Each process of an older version gets a grace period to finish its work. + The default is 60 seconds. A plugin can declare its own value in + `rpc.discover` (`x-sausage-grace`). Then the core stops the process. +3. When the old process of a service has ended, the core starts the service + again with the new version. +4. When the last process of an older version has ended, the core starts a + process of the new version and calls `rpc.retired {"previous": + ""}`. A plugin can use this hook to migrate its + state, or to write a marker that its other new processes wait for. The + result goes to the journal. + +During the grace period, an old process can call a method that a new +provider no longer has. + +### 3.8 Environment and network + +- `SAUSAGE_PID`: the process ID. +- `SAUSAGE_PLUGIN`: the plugin name. +- `SAUSAGE_PARENT`: the parent: `console`, `core`, or a process ID. + +Every process gets the network: outgoing HTTP and HTTPS through `wasi:http`, +and TCP, UDP, and DNS through `wasi:sockets`. + +A process that computes for a long time does not block other processes. The +core interrupts it every 10 milliseconds. + +### 3.9 Errors + +The standard JSON-RPC codes apply. The core adds these codes: + +| Code | Meaning | +| --- | --- | +| -32001 | There is no such plugin. | +| -32002 | The process ended without an answer. `data` has the exit status and the last journal lines. | +| -32003 | The activation was rejected. `data` has the reasons. | + +## 4. The filesystem + +Each process gets one preopen, `/`. The core resolves each path through a +mount table: + +| Path | What it is | Access | | --- | --- | --- | -| `spawn` | `actor`, `wasm`, `source`?, `built_by`? | test, then start a new actor | -| `upgrade` | `actor`, `wasm`, `source`?, `built_by`? | test, then swap the version: a running actor is stopped (normal shutdown) and starts the new version; a stopped actor gets the new version and stays stopped, with its mailbox untouched, until `start`. The answer has `running` | -| `rollback` | `actor`, `version` | go back to an older version; like `upgrade`, a stopped actor stays stopped. The answer has `running` | -| `stop`, `start` | `actor` | stop or start an actor | -| `remove` | `actor` | stop the actor (normal shutdown), then delete its whole folder `/actor//` (all versions, `data/`, `public/`, logs); answer `{"type": "removed", "actor"}`. Messages to it are then undeliverable, and the ID can be spawned again. An unknown actor or a native one (`broker`, `console`, `cargo`, `fs`) is an `error` | -| `subscribe`, `unsubscribe` | `actor` | receive (or not) what `actor` publishes | -| `tap`, `untap` | | receive (or not) a copy of every message the broker routes | -| `watch`, `unwatch` | `path` | receive (or not) `fs_changed` for every change of `path` or anything under it; answer `{"type": "watching" \| "unwatched", "path"}` with the path as an absolute world path | -| `exit` | | answer `exiting`, stop every actor, and end the `sausage` process with exit code 75 (a wrapper like `nix run .#loop` starts it again, updated) | +| `/` | the world folder | read-write | +| `/proc` | generated by the core | read-only | +| `/self` | the folder `/proc//`, in memory | read-write | +| `/run` | in memory, shared, empty at each host start | read-write | -**Headless.** `sausage run --headless [dir]` (or `sausage --headless`, or `SAUSAGE_HEADLESS=1`) turns the console input off: the host does not read stdin, so end of input (`/dev/null`) does not stop the world. `nix run .#loop -- --headless ` passes it through the loop. It runs until SIGTERM or SIGINT (stops every actor, exit code 0) or a broker `exit` request (exit code 75, as above). Without the flag, EOF on stdin or `/quit` ends the host. +A listing of `/` shows `proc`, `run`, and `self`. A rename across two of +these areas fails. -Paths are absolute world paths, or the sender's `/self/...`, `/public/...` or -`/data/...` aliases. Native senders (such as the console) can also supply -absolute host paths; apart from `/actor` and these aliases, -these retain their host meaning. - -**Taps.** After `tap`, the sender gets a copy of every message that the -broker routes (direct messages, replies, and publishes, between any actors, -including its own), for debugging, logging, or a live view of the world: - -```json -{"type": "tapped", "message": {"id": "m-17", "from": "console", "to": "agent", "reply_to": null, "body": {...}}, "ts": 1792132579067} +```text +/proc/ + core/ + PROTOCOL.md this file + graph.json plugins, imports, and the result of each check + graph.dot the same graph for Graphviz + locks.json held locks and waiting processes + plugins// + discover.json the rpc.discover document of the current plugin.wasm + client.rs the Rust code that import!("") generates + / + info.json plugin, version, method, params, parent, detached, start time, state + log the journal lines of this process + the files that the process writes to /self ``` -- `to` is `null` for a publish: one copy per publish, not per subscriber. -- `ts` is the time of routing, in milliseconds since the Unix epoch. -- Copies come from `broker`. They are not routed themselves, so they are - never tapped, and each message is tapped once per tapper. -- A copy that does not fit into the tapper's mailbox is dropped; routing - never waits for a tapper, and nobody gets `undeliverable` for a copy. -- Never answer a copy: the answer would be tapped again, forever. -- Like subscriptions, a tap is saved in `actor.json` (`"tapped": true`) and - survives restarts and upgrades. `untap` ends it. +Rules: -**Watches.** The host watches the whole world folder with the OS file -watcher: every change inside it counts, whoever makes it (actors, a 9P mount, -a shell or `git` on the host). Actor-served files (`/actor//fs/`) are not -on disk and do not count. After `watch {path}`, the sender gets for each -change of `path` or a path under it (`/actor/a` covers `/actor/a/x`, not -`/actor/ab`): +- A process writes its own folder only through `/self`. Writes through + `/proc` always fail. +- In a process folder, the names `info.json`, `log`, and `fs` belong to the + core. A write to `/self/info.json` fails. +- A read of a generated file returns a snapshot that ends. For example, + `/proc//log` returns the lines up to the time of the open. +- The folder of a process goes away when the process ends. Its files are in + memory. Large or persistent data belongs in `/state`. +- Do not send a `/self` path to another process. There it means a different + folder, like `localhost` on another machine. + +Principle: **inspect through files, act through calls.** + +## 5. Endpoints + +An endpoint is a JSON-RPC interface that a running process serves over TCP, +for example the control interface of a web server. + +- The process binds a free port on 127.0.0.1. When the listener is ready, it + writes an endpoint file in one atomic step (write a temporary file, then + rename it). The file has one line: the address, the owner process, and the + interface. Example: `tcp:127.0.0.1:41234 www-0197… control`. +- Usually the file is in `/self`. A service that others must find uses a + fixed path in `/run`, for example `/run/secrets`. +- A client reads the file and connects. Its first request is + `rpc.endpoint`, which returns `{"owner", "plugin", "interface"}`. If they + do not match the file, the client disconnects. So a stale file or a reused + port cannot connect a client to the wrong process. +- JSON-RPC runs on the connection, one message per line, in both directions. + Notifications of a handler (`core.notify`) go over the connection. +- A plugin declares its endpoint interfaces in `rpc.discover` + (`x-sausage-endpoints`). + +## 6. Locks + +- `core.lock {"path": "/state/notes/notes.json"}` returns `{"held": true}` + when the caller holds the lock. +- Each acquisition gets its own place in a FIFO queue. The same process + cannot take one lock twice: its second acquisition waits behind its first. +- With `"wait": false`, the call returns at once with `{"held": true}` or + `{"held": false}`. +- `core.unlock {path}` (a notification) releases the lock. +- When a process ends, the core releases all of its locks, also after a + crash or a kill. It waits until the last file operation of the process has + returned. So a write of a killed process cannot land after the next + process got the lock. +- Locks are advisory. The core does not block writes to the file: the path + is only a name. +- SQLite has no file locks in WASI. Hold the lock while the database is open. + +## 7. The journal + +- An SQLite file at `/state/core/journal.sqlite`. Only the core writes it. +- Each entry has a UUIDv7 (it sorts by time), the process ID, the plugin, the + stream (`stderr` or `core`), and the line. +- The core writes its own events too: start (with method and params), exit + status, stop, kill, upgrades, and the warning about discarded stdout. +- `core.log {plugin?, pid?, since?, until?, grep?, limit?}` queries the + journal. `/proc//log` shows the lines of one running process. + +## 8. Boot and services + +At start, the core starts each service in `/config/boot.json`, in order: ```json -{"type": "fs_changed", "path": "/actor/notes/public/list.md", "op": "write"} +[ + {"method": "secrets.serve", "params": {}}, + {"method": "www.start", "params": {"port": 8080}} +] ``` -- `op` is `write`, `create`, `remove`, or `rename` (then `to` is the new - path; a watcher of either path gets it). -- Changes of one path within about 100 ms become one event (a `write` after - a `create` stays a `create`). -- The host's temporary files (names starting with `.sausage-tmp-`) are - skipped; renaming one onto a file is a `write` of that file. -- Events come from `fs`. Like tap copies, they are not routed, a full - mailbox drops them, and they never block the writer. Never answer them. -- If the OS drops events, every watcher gets one - `{"type": "fs_changed", "path": "/", "op": "rescan"}`: anything may - have changed. -- A watch is saved in `actor.json` (`"watches": [...]`) and survives - restarts and upgrades. `unwatch` with the same path ends it. +- A service is a detached call with the parent `core`. The core waits for + its result before it starts the next one. The result shows that the + service is ready. +- If a service fails, the core writes the error to the journal and continues. +- After an upgrade of the plugin of a service, the core starts the service + again with the new version (section 3.7). +- The core does not restart other detached processes. +- A singleton is a service with a fixed endpoint in `/run`. Only the core + starts it, so two instances cannot start at the same time. -**Versions.** A new version (from `spawn`, `upgrade`, or a changed -`actor.wasm` on disk) is tested first, while the old version keeps running. -Only when its tests pass does the broker copy it to `versions//` -(`actor.wasm`, `source/`, `meta.json`) and switch to it. So `versions/` holds -exactly the versions that were installed, and instances always run from -there. If the tests fail, nothing changes, and the sender gets an `error` -with the reason. All versions share the actor's `data/` and `public/`: a -new version must read the data of the old one. +## 9. The core API -**Tests.** The broker runs the candidate on a copy of the actor's `data/` -and `public/`, without network. The tests are JSON files in -`/tests/*.json`, one scenario per file: +| Method | Params | Result | +| --- | --- | --- | +| `core.call` | `method`, `params`, `detach`? | the result of the method | +| `core.notify` | `name`, `params` | none (a notification) | +| `core.stop` | `pid` | `{}` when the process has ended | +| `core.kill` | `pid` | `{}` | +| `core.lock` | `path`, `wait`? | `{"held": bool}` | +| `core.unlock` | `path` | none (a notification) | +| `core.activate` | `plugin` and `wasm`, or `plugins: [{plugin, wasm}]`. `force`? | the new `rpc.discover` documents | +| `core.revert` | `plugin`, `force`? | like `core.activate` | +| `core.log` | filters (section 7) | journal entries | +| `core.exit` | none | `{}`. Then the host stops with exit code 75. | -```json -{ - "description": "add, then list", - "steps": [ - {"send": {"type": "add", "text": "milk"}, "expect": {"type": "added"}}, - {"send": {"type": "list"}, "expect": [{"type": "notes"}]} - ] -} -``` +Everything else is a file. -Each step sends `send` (from `from`, default `test`), then reads the next -line that the candidate writes for each `expect` entry, in order, within 10 -seconds. An entry matches when it is a subset of the message body. With -`to`, the message must also go to that recipient. Each file runs in a fresh -instance. A version without test files passes. +## 10. Interfaces -**Swap.** The broker holds the mailbox, stops the old instance, makes the -top-level `actor.wasm` and `source/` a copy of the new version, and starts -it. A stopped actor is not started: it keeps its mailbox and runs the new -version at the next `start`. A rollback does the same with the older version. +### 10.1 `rpc.discover` -**Crashes.** The broker restarts a crashed actor after 0.5 s, then doubles -the delay up to 30 s. A new version that crashes 3 times within 30 seconds -after an upgrade goes back to the previous version. The actor that asked for -the upgrade gets `{"type": "rolled_back", "from", "to", "reason", "last_exit"}`. +Each plugin answers `rpc.discover` with an OpenRPC 1.2 document. The schemas +are JSON Schema Draft 7, as OpenRPC requires. Shared schemas are in +`components.schemas`, and `$ref`s point to `#/components/schemas/`. +Params are by name. The document can have these extension fields: -**The folder is the truth.** At start, the broker boots every -`actor//actor.wasm`. Afterwards, the file watcher reports a new or -changed `actor//actor.wasm` (or a new folder `actor//`) once it is -quiet for about 100 ms: a new file becomes a new actor, a changed one an -upgrade, tested first with the tests in `actor//source/`. A file equal -to the installed version changes nothing. If its tests fail, the file stays -where it is, the reason goes to `stderr.log`, and the next change of the -file tries again. When the OS drops events, the broker checks every actor -folder again, as at start. +- `x-sausage-params` for each method: a `$ref` to the schema of the params + object. +- `x-sausage-endpoints`: the endpoint interfaces by name, each with its + `methods`. +- `x-sausage-imports`: for each provider, the part of its document that the + plugin uses. +- `x-sausage-grace`: the grace period in seconds. -## The `cargo` actor +The schema of params describes what the provider can read. The schema of a +result describes what the provider writes. Descriptions are part of the +interface: other agents read them to use it. -```json -{"type": "build", "source": "/actor/agent/public/drafts/notes"} -``` +The core starts a process only to call `rpc.discover`. Do not do expensive +work at start. -The build request may include `"skip_tests": true` to skip Rust unit tests. -By default, `cargo` first runs `cargo test` for the host target (not WASI), -so pure-logic unit tests in a `wasm32-wasip2` actor crate can run without a -WASI test runner. If tests fail or cannot compile, the build fails before the -WASI build. The reply carries `tests: {"passed": , "failed": , -"output": ""}`; counts sum Cargo's test-result -summaries, and a test command/compilation failure without a summary counts as -one failure. With `skip_tests: true`, counts are zero and output says tests -were skipped. +### 10.2 Compatibility -The reply is `{"type": "built", "wasm": "/actor/cargo/public/builds//.wasm", "built_by": "cargo", "tests": {...}, ...}` -or `{"type": "build_failed", "errors": "...", "tests": {...}}`. Pass -`wasm`, `source`, and `built_by` on to `spawn` or `upgrade`. +A provider can add methods, optional params, and result fields. For each +imported method: -Build scripts can compile C for `wasm32-wasip2`: the host sets -`CC_wasm32_wasip2` and `AR_wasm32_wasip2` to a WASI clang with wasi-libc, so -the `cc` crate (for example `rusqlite` with `bundled`) works. Its headers are -WASI 0.1: C code that needs sockets does not build. +1. The provider has the method. +2. Each params value that the consumer can send is valid for the provider. +3. Each result that the provider can return is valid for the consumer. -A Rust actor depends on the SDK with `sausage-actor = "0.1"`. `cargo` points -that dependency at the SDK of the host, which it publishes at -`/actor/cargo/public/sdk/`. +- The same rules apply to endpoint interfaces. +- There are no version numbers. This check replaces them. +- The check covers a fixed subset of JSON Schema: objects with `properties` + and `required`, the primitive types, arrays, optional and nullable values, + string enums, integer ranges, and `$ref`. A construct outside the subset + fails the check. +- The check covers calls, not state. Old and new processes share the files in + `/state`, and `core.revert` restores only code. A plugin must keep its + state format compatible, or migrate it in `rpc.retired`. + +### 10.3 Activation + +`core.activate {"plugin": "notes", "wasm": "/state/cargo/builds/7/notes.wasm"}`: + +1. The core copies the file. All further steps use this copy, even if the + file changes. +2. It starts the copy and calls `rpc.discover`. +3. It waits until it has the documents of all plugins (section 10.4). +4. It checks the imports of the new version against the current providers. +5. It checks each current consumer of the plugin against the new version. +6. If all checks pass, `plugin.wasm` becomes `plugin.prev.wasm`, and the copy + becomes `plugin.wasm`. Then the upgrade starts (section 3.7). + +If a check fails, nothing changes, and the error -32003 lists each problem, +with a counterexample where possible. + +- The core runs one activation at a time. +- `plugins: [...]` activates several plugins together. The checks then use + the new set. +- `force: true` skips steps 4 and 5. +- `core.revert {plugin}` activates `plugin.prev.wasm` in the same way, and + the current version becomes `plugin.prev.wasm`. +- A person can also copy a file to `plugin.wasm` by hand. That skips the + checks. + +### 10.4 The graph + +- The core keeps the `rpc.discover` documents of all plugins, by file hash. + At start, it gets them in the background. +- A file watcher on `/plugins/*/plugin.wasm` updates them, for example after + a copy by hand. `/proc/core/graph.json` then shows each broken import, and + the journal records it. +- An import of a plugin that does not exist is "missing". + +## 11. Building and tests + +The core has one native plugin: `cargo`. `cargo.build {"plugin": "notes"}` +builds `/plugins/notes/source/`: + +1. It copies the source once. The tests and the build both use this copy. +2. It runs the tests under WASI. +3. It builds the release `.wasm` for `wasm32-wasip2`. +4. It returns `{"wasm": "/state/cargo/builds//notes.wasm", "tests": + {"passed", "failed", "output"}, "warnings"}`. If a step fails, it returns + an error with the compiler or test output. + +- A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal + entries, like other calls. +- The dependency `sausage-plugin = "0.1"` points to the SDK copy at + `/plugins/cargo/sdk/`. +- Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2` + point to a WASI clang. +- The build does not activate the result. Call `core.activate` next. + +Tests: + +- Each test runs in its own instance, so a panic ends only its own test. +- The normal Rust test setup works: `#[test]`, `tests/*.rs`, name filters, + and `--ignored`. `#[should_panic]` cannot work, because WASI cannot catch + a panic. +- Each test gets an empty temporary world as `/`, the package folder at its + path, and its own `/proc`, `/self`, and `/run`. +- The test runner answers `core.lock`, `core.unlock`, `core.notify`, and + `core.log`. A call to a plugin fails: a test has no other plugins. diff --git a/README.md b/README.md index 5ab86b0..4a4775d 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ You can see how the sausage is made. -Sausage is a world of small programs ("actors") that you grow by talking to -it. Each actor is a WASI program with stdin and stdout. Actors send each other -JSON messages, one per line, through a broker. The agent is an actor too: it -writes new actors in Rust, builds them, tests them, starts them, and can -change its own code. +Sausage is a world of small WASI programs ("plugins") that you grow by +talking to it. A plugin is a `.wasm` file with JSON-RPC methods. Each call +to a plugin starts a new process. The agent is a plugin too: it writes new +plugins in Rust, builds them, tests them, activates them, and can change its +own code. ## Start @@ -29,8 +29,8 @@ nix run git+https://gitea.surma.technology/surma/sausage -- run # starts the `init` asks for the API shape (`openai-responses` or `anthropic-messages`), the base URL, the model, and the token. It reads `SAUSAGE_API`, `SAUSAGE_BASE_URL`, `SAUSAGE_MODEL`, and `SAUSAGE_API_TOKEN` first. The -values go to `world/actor/agent/data/config.json`. Both commands take another -folder as an argument, for example `... -- run my-world`. +values go to `world/config/agent.json`. Both commands take another folder as +an argument, for example `... -- run my-world`. To keep a world on the newest version, run it in a loop: @@ -39,60 +39,61 @@ nix run git+https://gitea.surma.technology/surma/sausage#loop # ./ nix run git+https://gitea.surma.technology/surma/sausage#loop -- my-world # an existing world ``` -The loop runs `nix run --refresh git+https://gitea.surma.technology/surma/sausage -- run my-world`. -When the host crashes or an actor sends `@broker {"type":"exit"}`, the loop -fetches the newest commit and starts it again. `/quit` ends the loop. Set -`SAUSAGE_FLAKE` to run another flake URL instead, for example your fork. +When the host crashes or a plugin calls `core.exit`, the loop fetches the +newest commit and starts it again. `quit` ends the loop. Set `SAUSAGE_FLAKE` +to run another flake URL instead, for example your fork. From a clone (`git clone https://gitea.surma.technology/surma/sausage`), the -same commands are `nix run`, `nix run -- init`, `nix run -- run` and +same commands are `nix run`, `nix run -- init`, `nix run -- run`, and `nix run .#loop`. ### Headless -`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off: the host -never reads stdin, so end of input does not stop the world. It runs until -SIGTERM/SIGINT (clean stop, exit code 0) or a broker `exit` request (exit -code 75, so the loop starts the newest version). Create the world first -(`init`), then pass the flag through the loop: +`--headless` (or `SAUSAGE_HEADLESS=1`) turns the console input off. The host +runs until SIGTERM or SIGINT (exit code 0) or until a plugin calls +`core.exit` (exit code 75, so the loop starts the newest version): ```sh nix run git+https://gitea.surma.technology/surma/sausage#loop -- --headless my-world ``` -In the console: +## The console -```text -write an actor "notes" that stores notes, and start it → text to the agent -@notes {"type":"add","text":"milk"} → a JSON message to an actor -@agent hello → text to an actor -/quit -``` +| Input | Effect | +| --- | --- | +| text | a message to the agent (`agent.chat`) | +| `. {json}` | a call. The console prints the result. | +| `ls [path]`, `cat ` | the filesystem as a process sees it, with `/proc` and `/run` | +| `methods ` | the methods of a plugin, with params and docs | +| `ps` | the running processes | +| `log [plugin or pid] [-n N]` | the journal | +| `activate `, `revert ` | activate a new version, or go back | +| `stop `, `kill ` | stop a process | +| `help`, `quit` | | ## The world folder -The folder is the source of truth. Every actor lives in `world/actor//`: - ```text -actor.wasm the current version -actor.json installed version, running, subscriptions -source/ the source of the current version -public/ files that the actor publishes -data/ actor state; owner-writable, world-readable via /actor//data/ (not private) -stderr.log what the actor and the broker write about it -versions// every installed version: actor.wasm, source/, meta.json +plugins//plugin.wasm the current version of a plugin +plugins//source/ its source +state// the state of a plugin, by convention +state/core/journal.sqlite the journal: stderr of all processes, and core events +config/boot.json the services that start with the world +config/agent.json the configuration of the agent ``` -To add or replace an actor by hand, put an `actor.wasm` (and its `source/`, -with the tests in `source/tests/`) into its folder; the host watches the -world folder and picks the file up once it stops changing. The broker tests -the file before it uses it. -If the tests fail, the old version keeps running, and `stderr.log` says why. +Each process also sees `/proc` (the processes and the state of the core), +`/self` (its own folder in `/proc`), and `/run` (shared files in memory). + +To replace a plugin by hand, copy a `.wasm` file to its `plugin.wasm`. That +skips the interface checks. `revert ` goes back to the version +before the last activation. ## More -- [PROTOCOL.md](PROTOCOL.md): messages, the filesystem, the broker, `cargo`. - Read this to write an actor in any language. -- [`sdk/`](sdk/): `sausage-actor`, the SDK for Rust actors. -- [`examples/notes`](examples/notes): a small actor with a test. +- [PROTOCOL.md](PROTOCOL.md): calls, processes, the filesystem, locks, + interfaces, and `cargo`. Read this to write a plugin in any language. +- [DECISIONS.md](DECISIONS.md): why Sausage is the way it is. +- [`sdk/`](sdk/): `sausage-plugin`, the SDK for Rust plugins. +- [`examples/`](examples/): small plugins. - [AGENTS.md](AGENTS.md): how to work on Sausage itself.