surma/sausage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Trim the harness to what it needs to keep going

Browse Source 
Host:
- test first: a new version is tested where it lies; only an installed
  version goes to versions/, and instances run from there
- data/ and public/ are shared by all versions; no snapshots; rollback
  restores code only
- no re-delivery, status.json, list, remove/archive, smoke test, data/
  hiding, or stdin timeout; rollback needs a version
- supervisor.rs split out; one place for installing a version
- init only writes the agent folder; run installs it like any actor
- console: @actor, text to the agent, /help, /quit

Agent:
- every text goes out as a broadcast; 5-line prompt; 5 tools that describe
  themselves; turn.json only marks an active turn

Docs: PROTOCOL.md, README.md, DECISIONS.md, AGENTS.md rewritten.
Examples: only notes and notes-v2.
This commit is contained in:
surma
2026-10-15 13:35:45 +01:00
parent cee862275c
commit 11144b947e
35 changed files with 909 additions and 2038 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+13 -8
View File
@@ -11,26 +11,31 @@ Keep it that way.
3. **Readable.** A new reader must understand a file without the others. 3. **Readable.** A new reader must understand a file without the others.
Each module starts with a doc comment that says what it owns. Each module starts with a doc comment that says what it owns.
4. **Efficient** on the paths that run per message. 4. **Efficient** on the paths that run per message.
5. **Raw.** Sausage has no permissions, sandboxing beyond WASI, limits, or 5. **Raw and minimal.** Sausage has no permissions, sandboxing beyond WASI,
authentication. Do not add them to the core. People who want them can limits, or authentication. Do not add them to the core. Add nothing to
build them as actors or as a fork. the host that an actor (or the agent, by changing itself) could do.
## Layout ## Layout
- `cli/`: the host (`sausage` binary and library). `broker.rs` routes and - `cli/`: the host (`sausage` binary and library).
supervises, `fs.rs` is the filesystem layer, `runtime.rs` runs Wasmtime, - `lib.rs`: `Host`, which starts and stops a world.
`candidate.rs` tests new versions, `cargo_actor.rs` builds Rust actors. - `broker.rs`: routing, mailboxes, the version store, broker requests.
- `supervisor.rs`: runs one WASM actor (restarts, swaps, rollbacks).
- `runtime.rs`: Wasmtime instances. `fs.rs`: commit on close.
- `candidate.rs`: tests a new version. `cargo_actor.rs`: builds Rust.
- `world.rs`: the on-disk layout. `console.rs`: the terminal.
- `sdk/`: `sausage-actor`, the SDK for Rust actors. Never published: the - `sdk/`: `sausage-actor`, the SDK for Rust actors. Never published: the
`cargo` actor points `sausage-actor = "0.1"` at its local copy. `cargo` actor points `sausage-actor = "0.1"` at its local copy.
- `agent/`: the agent actor. `sausage init` embeds and builds it. - `agent/`: the agent actor. `sausage init` embeds and builds it.
- `examples/`: small actors. They are not workspace members. - `examples/`: small actors. They are not workspace members.
- `PROTOCOL.md`: the contract between the host and the actors. Change it in - `PROTOCOL.md`: the contract between the host and the actors. Change it in
the same commit as the behavior. the same commit as the behavior. `DECISIONS.md` says why; when the two
disagree, `PROTOCOL.md` and the code win.
## Checks ## Checks
```sh ```sh
nix develop -c cargo test -p sausage -p sausage-actor nix develop -c cargo test -p sausage -p sausage-actor -p agent
nix develop -c cargo build --release -p agent --target wasm32-wasip2 nix develop -c cargo build --release -p agent --target wasm32-wasip2
nix develop -c cargo clippy --workspace nix develop -c cargo clippy --workspace
``` ```
DECISIONS.md
+44 -145
View File
@@ -1,154 +1,53 @@
# Decisions # Decisions
This file lists the design decisions of the spike. **(you)** marks a decision Why Sausage is the way it is. `PROTOCOL.md` and the code own the details;
that Surma made. **(spike)** marks a decision that Scout made alone during the when this file disagrees with them, they win.
night and that needs review.
## Architecture ## The shape
- **(you)** Actors are WASI commands (`wasm32-wasip2`). They use stdin, stdout, - **Actors are WASI commands, not components with interfaces.** stdin,
files, and HTTP. No WIT interfaces, no component composition. stdout, files, and HTTP are enough, and every language can do them.
- **(you)** Actors exchange newline-delimited JSON through the host, which is - **Messages are JSON lines through a broker, fire and forget.** Like
the broker. No built-in request/response. A reply is a message with Erlang: no acknowledgments, no request/response. A reply is a message with
`reply_to`. Delivery is at most once. `reply_to`. Sausage is not a reliable message bus.
- **(you)** The broker accepts only `.wasm` files. It knows no source language. - **The broker accepts only `.wasm` files.** It knows no source language.
- **(you)** `cargo` is a built-in Rust compiler actor. **(spike)** It is native `cargo` is one compiler actor; others can add more languages.
(host code), because cargo cannot run inside WASM. - **The world folder is the source of truth.** Everything about an actor is
- **(you)** The agent is a normal WASM actor with the fixed ID `agent`. It can in `actor/<id>/`. Copy a folder into another world, and it runs there.
read and upgrade its own source. - **The agent is a normal WASM actor.** It can read and upgrade its own
- **(spike)** Three native actors: `broker`, `console`, `cargo`. source, and change how it does inference.
- **(you)** One Cargo workspace with two crates: `cli` (binary `sausage`) and
`agent`. **(spike)** Example actors are in `examples/`, outside the workspace.
- **(you)** `sausage init` builds the agent and asks for the API shape, the
endpoint, the model, and the token. **(spike)** The token can also come from
`SAUSAGE_API_TOKEN`. The source contains no endpoint and no token.
## Messages ## Minimal on purpose
- **(spike)** Stdin line: `{"id", "from", "reply_to"?, "body"}`. The harness does only what it needs to keep going. Everything else can be an
Stdout line: `{"to"?, "id"?, "reply_to"?, "body"}`. actor, or a change that the agent makes to itself.
- **(spike)** The broker sets `from`. An actor can choose its own `id`. The
broker then uses `"<sender>/<id>"`, so IDs cannot collide or be faked.
Without an `id`, the broker assigns `m-<n>`.
- **(you)** Without `to`, a message goes to the subscribers of the sender.
**(spike)** `subscribe` / `unsubscribe` messages to the broker. The registry
stores subscriptions, so they survive restarts and upgrades. `init`
subscribes `console` to `agent`.
- **(spike)** Errors from routing come back as
`{"type": "undeliverable", "reason", "to"}` from `broker`.
- **(spike)** Lifecycle: `{"type": "shutdown"}` from `broker`, then EOF on stdin.
After 5 seconds the broker kills the instance.
- **(spike)** Lines on stdout that are not JSON objects with `body` go to the
actor's `stderr.log` with a note.
## Filesystem - **The host is not self-modifiable; actors are.** So the host keeps only
what an actor cannot do for itself: routing, running instances, versions,
tests before an install, and the rollback of a version that crash-loops.
Without that rollback, a broken self-upgrade would end the agent.
- **No permissions, no limits, no hiding.** Any actor can message, upgrade,
or stop any actor, read every folder, and use HTTP. People who want rules
can build them.
- **No re-delivery and no state snapshots.** A crash loses the message, and
all versions of an actor share `data/` and `public/`. A rollback restores
code, not data.
- **The agent is dumb.** It has a short prompt, a few tools, an append-only
history that it sends in full, and no compaction. When the context is too
large, it stops sending the oldest turns, so it never gets stuck.
- **(you)** `/data` private, `/public` own public files, ## Small choices
`/actor/<id>/public` and `/actor/<id>/source` read-only for everyone.
- **(spike)** The broker also writes `/actor/<id>/status.json` and
`/actor/<id>/stderr.log`. `ls /actor` is the registry.
- **(you)** Commit on close: a write becomes visible only when the writer
closes the file. **(spike)** This also applies to `/data`. Implementation: a
wrapper around the `wasmtime-wasi` filesystem. An open for write goes to a
hidden `.sausage-tmp-*` file next to the target. The descriptor drop renames
it into place. Details:
- A normal exit commits every open file (process exit closes files).
- A trap or kill discards uncommitted writes.
- An append or an edit without truncate copies the old content first.
- Other descriptors see the old content until the close.
- Directory operations take effect immediately.
- The layer hides `.sausage-tmp-*` names from listings, `stat`, `open`,
`rename`, and `unlink`.
- Symlinks are disabled (they would break the path tracking).
- No `fsync`. A power loss can lose committed data.
- **(spike)** Raw sockets are not available (no network permission), but the
`wasi:sockets` interfaces are linked, so programs that import them still
start.
## Broker requests - **Commit on close** for new and truncated files, so readers never see half
a file, in any language. Appends write in place, so logs stay cheap.
- **(spike)** `spawn`, `upgrade`, `rollback`, `stop`, `start`, `remove`, - **Test first, then switch.** A new version is tested where it lies. Only
`subscribe`, `unsubscribe`, `list`. a version that passes goes to `versions/`, and the old one runs until
- **(you)** `spawn` and `upgrade` take `wasm`, plus optional `source` and then. `versions/` keeps every version that was installed, so a rollback
`built_by` metadata. **(spike)** The broker copies the `.wasm` file and the can always go back. A file that fails stays where it was put.
source folder into `broker/versions/<id>/<n>/`, with `meta.json` (paths, - **Tests are JSON transcripts** next to the source. They run on a copy of
builder, requester, time). the actor's state, without HTTP, so a test cannot send real requests.
- **(spike)** Path resolution: `/actor/...` is global, and `/public/...` and - **`broker` and `cargo` never answer a reply,** so no two actors answer
`/data/...` belong to the sender. `..` is rejected. Native senders (the each other forever. The agent starts a turn only for text that is not a
console) can give absolute host paths. reply, for the same reason.
- **(you)** Anyone can upgrade anyone. No permissions. - **The SDK is not published.** `cargo` points `sausage-actor = "0.1"` at its
- **(spike)** Actor IDs: `a-z`, `0-9`, `-`, `_`, at most 64 characters. own copy, which it also shows in `/actor/cargo/public/sdk/`.
- **(spike)** `remove` archives every folder of the actor to `archive/`. Nothing
is deleted.
## Upgrades, tests, rollback
- **(spike)** The tests are message transcripts in `source/tests/*.jsonl`:
`{"send": body}` and `{"expect": subset-of-body, "to"?}`. Each file runs in
a fresh candidate on a copy of the actor's `/data` and `/public`. The
candidate has no HTTP access. An `expect` waits up to 10 s. Without tests,
the candidate must stay up for 2 s (smoke test).
- **(spike)** Swap: hold the mailbox, stop the old instance gracefully,
snapshot `/data` + `/public` into the old version's folder, install the new
source in `/actor/<id>/source`, start the new instance, release the mailbox.
- **(spike)** Restart after a crash: backoff 0.5 s × 2ⁿ, at most 30 s. The
counter resets after 60 s of uptime.
- **(spike)** Parked message: when an instance traps, the broker keeps the
last delivered message. It delivers that message again after the next
upgrade, not after a plain restart (that would loop).
- **(spike)** Crash loop: 3 crashes within 30 s after an upgrade → automatic
rollback to the previous version, with its data snapshot. The broker
informs the actor that requested the upgrade.
- **(spike)** `rollback` without `version` goes to the highest older version.
It restores that version's snapshot, and it first snapshots the current
state, so a roll-forward is possible.
- **(spike)** Versions are per actor, not global.
## Limits
- **(spike)** Mailbox: 1,000 messages. When it is full, the sender gets
`undeliverable`.
- **(spike)** The broker waits 30 s when it writes to an actor's stdin. Then it
drops the message and logs it.
- **(spike)** Time slices: the epoch advances every 10 ms, and each guest
yields at each epoch.
- **(spike)** One `sausage run` per world (file lock on `world/.lock`).
- **(spike)** HTTP for all running actors, with no allow-list yet.
## Agent
- **(you)** API shapes: OpenAI Responses and Anthropic Messages, chosen by
`"api"` in `/data/config.json`.
- **(spike)** Config fields: `api`, `base_url`, `model`, `token`, optional
`reasoning_effort` and `max_tokens` (16,000). OpenAI: `POST
{base_url}/v1/responses` with `store: false` and encrypted reasoning passed
back. Anthropic: `POST {base_url}/v1/messages` with `x-api-key`.
- **(spike)** One shared conversation in `/data/history.json`, in a neutral
format, so the API shape can change. The history is limited to 600 entries,
with the cut at a user message. There is no compaction yet.
- **(spike)** A turn starts only for `{"type": "text"}` messages *without*
`reply_to`. Replies never start turns, so two actors cannot keep the agent
in an endless loop. Other messages are collected and shown at the next
turn.
- **(spike)** Tools: `send_message` (with optional wait for the reply),
`wait_for_messages`, `read_file`, `list_dir`, `write_file`, `copy_dir`,
`remove_path`. At most 40 steps per turn. Tool output is cut at 30 KB.
- **(spike)** The final text of a turn goes to the sender, as a reply to the
message that started the turn.
- **(spike)** A shutdown during a turn saves the turn to `/data/turn.json`. The
next start (for example after a self-upgrade) resumes the turn and answers
the original message.
## Console
- **(spike)** `@<actor> <json|text>`, plain text to `agent`, `/deploy`,
`/list`, `/sleep`, `/await`, `/help`, `/quit`. `/sleep` and `/await` exist
for scripted tests.
## Known risks (accepted for now)
- Build scripts and proc macros run as native code in the host environment.
- Any actor can upgrade the agent with code that reads the token.
- Any actor can message, stop, upgrade, or remove any actor.
- A full copy of `/data` and `/public` at every upgrade is slow for large data.
- Mailboxes are in memory. A host crash loses queued messages.
PROTOCOL.md
+57 -48
View File
@@ -1,7 +1,7 @@
# The Sausage protocol # The Sausage protocol
This document is the contract between the host and the actors. An actor in This is the contract between the host and the actors. An actor in any
any language that follows it works in Sausage. The host publishes this file at language that follows it works in Sausage. The host publishes this file at
`/actor/broker/public/PROTOCOL.md`, so every actor can read it. `/actor/broker/public/PROTOCOL.md`, so every actor can read it.
## Actors ## Actors
@@ -34,19 +34,21 @@ A line on **stdout**:
``` ```
- `body` is required. By convention it is an object with a `"type"`. - `body` is required. By convention it is an object with a `"type"`.
Text for people is `{"type": "text", "text": "..."}`.
- `to` is the recipient. Without `to`, the broker publishes the message to - `to` is the recipient. Without `to`, the broker publishes the message to
every subscriber of the sender. every subscriber of the sender.
- `id` is optional. If the actor sets it, the message ID is `"<sender>/<id>"`. - `id` is optional. If the actor sets it, the message ID is `"<sender>/<id>"`.
Otherwise, the broker assigns `m-<n>`. The broker always sets `from`. Otherwise, the broker assigns `m-<n>`. The broker always sets `from`.
- `reply_to` marks a reply. There is no request/response: a reply is a new - `reply_to` marks a reply: its value is the ID of the message it answers.
message whose `reply_to` is the ID of the message it answers. There is no request/response. A reply is a new message.
- Delivery is at most once. Each actor has a mailbox of 1,000 messages. The - **Never answer a reply,** or two actors can answer each other forever.
mailbox keeps messages while the actor restarts or is upgraded. `broker` and `cargo` ignore messages that have a `reply_to`.
- If the broker cannot deliver a message, the sender gets - Messages are fire and forget: delivery is at most once. Each actor has a
mailbox of 1,000 messages, which keeps messages while the actor restarts
or is upgraded. If the broker cannot deliver a message, the sender gets
`{"type": "undeliverable", "reason": "...", "to": "..."}` from `broker`. `{"type": "undeliverable", "reason": "...", "to": "..."}` from `broker`.
- An actor must keep reading stdin. If a message waits 30 seconds for the - An actor must keep reading stdin. If the broker must stop or upgrade an
actor to read it, the broker kills and restarts the actor, and the sender actor while a message waits for it to read stdin, it kills the actor.
gets `undeliverable`.
Before the broker stops an instance, it sends Before the broker stops an instance, it sends
`{"id": "broker/shutdown", "from": "broker", "body": {"type": "shutdown"}}` `{"id": "broker/shutdown", "from": "broker", "body": {"type": "shutdown"}}`
@@ -59,13 +61,13 @@ Each instance sees three folders:
| Path | Access | Contents | | Path | Access | Contents |
| --- | --- | --- | | --- | --- | --- |
| `/data` | read-write | private state of the actor | | `/data` | read-write | the actor's state |
| `/public` | read-write | files that the actor publishes | | `/public` | read-write | the files that the actor publishes |
| `/actor` | read-only | the `actor/` folder of the world | | `/actor` | read-only | the `actor/` folder of the world |
`/actor/<id>/` contains `actor.wasm`, `actor.json`, `source/`, `public/`, `/actor/<id>/` holds everything about an actor: `actor.wasm`, `actor.json`
`status.json`, `stderr.log`, and `versions/<n>/` of every actor. The `data/` (installed version, running, subscriptions), `source/`, `public/`, `data/`,
folders (and the data snapshots in `versions/`) are hidden. `stderr.log`, and `versions/<n>/`. `ls /actor` lists all actors.
A new or truncated file becomes visible when the writer closes it. Before A new or truncated file becomes visible when the writer closes it. Before
that, readers see the old file or no file. If the instance crashes, the write that, readers see the old file or no file. If the instance crashes, the write
@@ -77,31 +79,34 @@ message.
## The broker ## The broker
Requests to `broker` (the reply has the same `reply_to`): Requests to `broker`. The answer is a message whose `reply_to` is the ID of
the request: the request's `type` in past tense (`spawned`, `upgraded`,
`rolled_back`, `stopped`, `started`, `subscribed`, `unsubscribed`,
`scanned`), or `{"type": "error", "reason": "..."}`.
| `type` | Fields | Effect | | `type` | Fields | Effect |
| --- | --- | --- | | --- | --- | --- |
| `spawn` | `actor`, `wasm`, `source`?, `built_by`? | test, then start a new actor | | `spawn` | `actor`, `wasm`, `source`?, `built_by`? | test, then start a new actor |
| `upgrade` | `actor`, `wasm`, `source`?, `built_by`? | test, then swap the version | | `upgrade` | `actor`, `wasm`, `source`?, `built_by`? | test, then swap the version |
| `rollback` | `actor`, `version`? | go back to an older version and its data | | `rollback` | `actor`, `version` | go back to an older version |
| `stop`, `start` | `actor` | stop or start an actor | | `stop`, `start` | `actor` | stop or start an actor |
| `remove` | `actor` | move the actor's folder to `archive/` |
| `subscribe`, `unsubscribe` | `actor` | receive (or not) what `actor` publishes | | `subscribe`, `unsubscribe` | `actor` | receive (or not) what `actor` publishes |
| `scan` | | boot new or changed `actor.wasm` files | | `scan` | | boot new or changed `actor.wasm` files |
| `list` | | all actors and their state |
Paths: `/actor/...`, `/public/...` (the sender's), or `/data/...` (the Paths are `/actor/...`, or the sender's `/public/...` or `/data/...`.
sender's).
**Versions.** `versions/<n>/` keeps every version that the broker saw: **Versions.** A new version (from `spawn`, `upgrade`, or a changed
`actor.wasm`, `source/`, and `meta.json`. A version that the broker refused `actor.wasm` on disk) is tested first, while the old version keeps running.
has `"rejected"` in its `meta.json`. Only when its tests pass does the broker copy it to `versions/<n>/`
(`actor.wasm`, `source/`, `meta.json`) and switch to it. So `versions/` holds
exactly the versions that were installed, and instances always run from
there. If the tests fail, nothing changes, and the sender gets an `error`
with the reason. All versions share the actor's `data/` and `public/`: a
new version must read the data of the old one.
**Tests.** Before a spawn or an upgrade, the broker runs the candidate on a **Tests.** The broker runs the candidate on a copy of the actor's `data/`
copy of the actor's `/data` and `/public` (empty for a new actor), without and `public/`, without HTTP. The tests are JSON files in
HTTP. The candidate sees its future `SAUSAGE_VERSION` and `<source>/tests/*.json`, one scenario per file:
`SAUSAGE_CANDIDATE=1`. The tests are JSON
files in `source/tests/*.json`:
```json ```json
{ {
@@ -113,25 +118,26 @@ files in `source/tests/*.json`:
} }
``` ```
An `expect` entry matches when it is a subset of the reply body. A step can Each step sends `send` (from `from`, default `test`), then reads the next
also have `from` (the sender, default `test`) and `to` (the expected line that the candidate writes for each `expect` entry, in order, within 10
recipient). Each file runs in a fresh instance. Without test files, the seconds. An entry matches when it is a subset of the message body. With
candidate must stay up for 2 seconds. `to`, the message must also go to that recipient. Each file runs in a fresh
instance. A version without test files passes.
**Swap.** The broker holds the mailbox, stops the old instance, saves a **Swap.** The broker holds the mailbox, stops the old instance, makes the
snapshot of `/data` and `/public` in the old version's folder, installs the new top-level `actor.wasm` and `source/` a copy of the new version, and starts
files, and starts the new instance. it. A rollback does the same with the older version.
**Crashes.** The broker restarts a crashed actor (0.5 s, then doubling, at **Crashes.** The broker restarts a crashed actor after 0.5 s, then doubles
most 30 s). It keeps the message that the actor got just before the crash, the delay up to 30 s. A new version that crashes 3 times within 30 seconds
and delivers it again after the next upgrade. A new version that crashes 3 after an upgrade goes back to the previous version. The actor that asked for
times within 30 seconds after an upgrade goes back to the previous version, the upgrade gets `{"type": "rolled_back", "from", "to", "reason", "last_exit"}`.
with its data. The actor that asked for the upgrade gets `rolled_back`.
**The folder is the truth.** At start, and on `scan`, the broker boots every **The folder is the truth.** At start, and on `scan`, the broker boots every
`actor/<id>/actor.wasm`. If the file differs from the installed version, the `actor/<id>/actor.wasm`. If the file differs from the installed version, it
broker copies it to `versions/<n>/` and upgrades to it. If the upgrade fails, is a new version, tested with the tests in `actor/<id>/source/`. If its
the installed files come back. tests fail, the file stays where it is, the reason goes to `stderr.log`, and
the next scan tries again.
## The `cargo` actor ## The `cargo` actor
@@ -139,7 +145,10 @@ the installed files come back.
{"type": "build", "source": "/actor/agent/public/drafts/notes"} {"type": "build", "source": "/actor/agent/public/drafts/notes"}
``` ```
The reply is `{"type": "built", "wasm": "/actor/cargo/public/builds/<n>/<name>.wasm", ...}` The reply is `{"type": "built", "wasm": "/actor/cargo/public/builds/<n>/<name>.wasm", "built_by": "cargo", ...}`
or `{"type": "build_failed", "errors": "..."}`. A Rust actor depends on the SDK or `{"type": "build_failed", "errors": "..."}`. Pass `wasm`, `source`, and
with `sausage-actor = "0.1"`. `cargo` points that dependency at the SDK of the `built_by` on to `spawn` or `upgrade`.
host.
A Rust actor depends on the SDK with `sausage-actor = "0.1"`. `cargo` points
that dependency at the SDK of the host, which it publishes at
`/actor/cargo/public/sdk/`.
README.md
+40 -45
View File
@@ -2,64 +2,59 @@
You can see how the sausage is made. You can see how the sausage is made.
Sausage is a system of actors that you grow by talking to it. Every actor is a Sausage is a world of small programs ("actors") that you grow by talking to
WASI program (a `.wasm` file) with stdin and stdout. Actors send each other it. Each actor is a WASI program with stdin and stdout. Actors send each other
newline-delimited JSON messages through a broker. The agent is an actor too: JSON messages, one per line, through a broker. The agent is an actor too: it
it does inference over HTTP, writes new actors, and can change its own code. writes new actors in Rust, builds them, tests them, starts them, and can
change its own code.
Inspired by [Jiti](https://ghuntley.com/lisp/), but with WASM instead of a ## Start
Lisp image.
## Quick start You need [Nix](https://nixos.org/download) with flakes.
```sh ```sh
nix develop nix run -- init # creates ./world and builds the agent (a few minutes)
cargo build --release nix run -- run # starts the world and a console
export SAUSAGE_API_TOKEN=... # or type it at the prompt
./target/release/sausage init world --api openai-responses \
--base-url https://api.example.com --model some-model
./target/release/sausage run world
``` ```
`init` builds the agent (the first build takes a few minutes) and writes its `init` asks for the API shape (`openai-responses` or `anthropic-messages`),
configuration to `world/private/agent/data/config.json`. The source contains the base URL, the model, and the token. It reads `SAUSAGE_API`,
no endpoint and no token. `SAUSAGE_BASE_URL`, `SAUSAGE_MODEL`, and `SAUSAGE_API_TOKEN` first. The
values go to `world/actor/agent/data/config.json`. Both commands take another
folder as an argument, for example `nix run -- run my-world`.
In the console: In the console:
```text ```text
hello → text to the agent write an actor "notes" that stores notes, and start it → text to the agent
@notes {"type":"add","text":"milk"} → a JSON message to an actor @notes {"type":"add","text":"milk"} → a JSON message to an actor
@agent what actors exist? → text to an actor @agent hello → text to an actor
/deploy notes examples/notes → build with cargo, then spawn or upgrade
/list → list actors
/await agent 300 → wait for the next message (scripts)
/quit /quit
``` ```
## Concepts ## The world folder
- **Actor**: a `wasi:cli/command` program. It reads messages on stdin and The folder is the source of truth. Every actor lives in `world/actor/<id>/`:
writes messages on stdout. Logs go to stderr.
- **Message**: `{"id", "from", "reply_to"?, "body"}` on stdin;
`{"to"?, "id"?, "reply_to"?, "body"}` on stdout. No `to` = publish to
subscribers. There is no request/response: a reply is a new message with
`reply_to`. Delivery is at most once.
- **Broker**: routes messages, owns mailboxes, and runs actors. It accepts
only `.wasm` files: `spawn`, `upgrade`, `rollback`, `stop`, `start`,
`remove`, `subscribe`, `unsubscribe`, `list`.
- **cargo**: the built-in Rust compiler actor. `build` a source folder, get a
`.wasm` path back. Other actors can add other languages.
- **Filesystem** of each actor:
- `/data`: private state
- `/public`: its public files
- `/actor/<id>/{public,source,status.json,stderr.log}`: every actor, read-only
A write becomes visible when the writer closes the file (commit on close). ```text
- **Upgrade**: the broker tests the candidate on a copy of `/data` and actor.wasm the current version
`/public` (transcripts in `source/tests/*.jsonl`). Then it holds new actor.json installed version, running, subscriptions
messages, stops the old version, starts the new one, and delivers the held source/ the source of the current version
messages. A new version that crashes 3 times in 30 seconds is rolled back. public/ files that the actor publishes
data/ the actor's state
stderr.log what the actor and the broker write about it
versions/<n>/ every installed version: actor.wasm, source/, meta.json
```
See [DECISIONS.md](DECISIONS.md) for every design decision and To add or replace an actor by hand, put an `actor.wasm` (and its `source/`,
[SPIKE.md](SPIKE.md) for the spike results. with the tests in `source/tests/`) into its folder. Then restart the world or
send `@broker {"type":"scan"}`. The broker tests the file before it uses it.
If the tests fail, the old version keeps running, and `stderr.log` says why.
## More
- [PROTOCOL.md](PROTOCOL.md): messages, the filesystem, the broker, `cargo`.
Read this to write an actor in any language.
- [`sdk/`](sdk/): `sausage-actor`, the SDK for Rust actors.
- [`examples/notes`](examples/notes): a small actor with a test.
- [AGENTS.md](AGENTS.md): how to work on Sausage itself.
agent/src/main.rs
+69 -194
View File
@@ -3,8 +3,7 @@
mod llm; mod llm;
use llm::{Config, Entry, Tool}; use llm::{Config, Entry, Tool};
use sausage_actor::{Inbox, Message, Value, json, reply, send}; use sausage_actor::{Inbox, Message, Value, json, publish, send};
use serde::{Deserialize, Serialize};
use std::collections::{HashSet, VecDeque}; use std::collections::{HashSet, VecDeque};
use std::io::Write; use std::io::Write;
use wstd::future::FutureExt; use wstd::future::FutureExt;
@@ -24,29 +23,20 @@ fn tools() -> Vec<Tool> {
vec![ vec![
Tool { Tool {
name: "send_message", name: "send_message",
description: "Send a message to an actor. With wait_seconds > 0, wait for its reply. Returns the message ID, the reply, and other messages that arrived meanwhile.", description: "Send an actor a JSON body with a `type`. `broker` and `cargo` are built-in actors; see `/actor/broker/public/PROTOCOL.md`. With `wait_seconds`, the reply's `reply_to` matches the returned ID. Returns the ID, reply, and other messages received.",
parameters: json!({ parameters: json!({
"type": "object", "type": "object",
"properties": { "properties": {
"to": {"type": "string", "description": "Actor ID"}, "to": {"type": "string", "description": "Actor ID"},
"body": {"type": "object", "description": "Message body. Must have a type field."}, "body": {"type": "object", "description": "JSON object with a type field"},
"wait_seconds": {"type": "number", "description": "Seconds to wait for the reply. 0 = do not wait."} "wait_seconds": {"type": "number", "description": "Seconds to wait for a reply. Default 0."}
}, },
"required": ["to", "body"] "required": ["to", "body"]
}), }),
}, },
Tool {
name: "wait_for_messages",
description: "Wait up to seconds for messages to arrive and return them.",
parameters: json!({
"type": "object",
"properties": {"seconds": {"type": "number"}},
"required": ["seconds"]
}),
},
Tool { Tool {
name: "read_file", name: "read_file",
description: "Read a text file. Paths: /data/..., /public/..., /actor/<id>/...", description: "Read a text file under `/data`, `/public`, or `/actor/<id>`. Reads all bytes by default; `offset` and `max_bytes` are optional.",
parameters: json!({ parameters: json!({
"type": "object", "type": "object",
"properties": { "properties": {
@@ -57,34 +47,29 @@ fn tools() -> Vec<Tool> {
"required": ["path"] "required": ["path"]
}), }),
}, },
Tool {
name: "list_dir",
description: "List a folder.",
parameters: json!({"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}),
},
Tool { Tool {
name: "write_file", name: "write_file",
description: "Write a text file under /data or /public. Creates parent folders.", description: "Write text under `/data` or `/public`. Parent folders are created.",
parameters: json!({ parameters: json!({
"type": "object", "type": "object",
"properties": {"path": {"type": "string"}, "content": {"type": "string"}}, "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
"required": ["path", "content"] "required": ["path", "content"]
}), }),
}, },
Tool {
name: "list_dir",
description: "List entries in a directory under `/data`, `/public`, or `/actor/<id>`. Returns each name, type, and size.",
parameters: json!({"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}),
},
Tool { Tool {
name: "copy_dir", name: "copy_dir",
description: "Copy a folder tree, for example /actor/<id>/source to /public/drafts/<id>. Skips target/ folders.", description: "Copy a directory tree, for example `/actor/<id>/source` to `/public/drafts/<id>`. Skips `target/` folders.",
parameters: json!({ parameters: json!({
"type": "object", "type": "object",
"properties": {"from": {"type": "string"}, "to": {"type": "string"}}, "properties": {"from": {"type": "string"}, "to": {"type": "string"}},
"required": ["from", "to"] "required": ["from", "to"]
}), }),
}, },
Tool {
name: "remove_path",
description: "Remove a file or folder under /data or /public.",
parameters: json!({"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}),
},
] ]
} }
@@ -108,19 +93,13 @@ fn copy_tree(from: &str, to: &str) -> std::io::Result<u64> {
Ok(count) Ok(count)
} }
#[derive(Default, Deserialize, Serialize)]
#[serde(default)]
struct TurnState {
active: Option<Message>,
queue: VecDeque<Message>,
notes: Vec<Message>,
}
struct Agent { struct Agent {
inbox: Inbox, inbox: Inbox,
history: Vec<Entry>, history: Vec<Entry>,
history_skip: usize, history_skip: usize,
pending: TurnState, active: bool,
queue: VecDeque<Message>,
notes: Vec<Message>,
shutdown: bool, shutdown: bool,
} }
@@ -135,33 +114,19 @@ impl Agent {
.ok() .ok()
.and_then(|skip| skip.trim().parse::<usize>().ok()) .and_then(|skip| skip.trim().parse::<usize>().ok())
.unwrap_or(0); .unwrap_or(0);
let pending = std::fs::read_to_string(TURN) let active = std::fs::read_to_string(TURN)
.ok() .ok()
.and_then(|contents| { .and_then(|contents| serde_json::from_str::<bool>(&contents).ok())
let value: Value = serde_json::from_str(&contents).ok()?; .unwrap_or(false);
let mut pending = serde_json::from_value::<TurnState>(value.clone()).ok()?; Self {
if pending.active.is_none() {
if let (Some(id), Some(from)) = (value["id"].as_str(), value["from"].as_str()) {
pending.active = Some(Message {
id: id.to_string(),
from: from.to_string(),
reply_to: None,
body: Value::Null,
});
}
}
Some(pending)
})
.unwrap_or_default();
let agent = Self {
inbox: Inbox::start(), inbox: Inbox::start(),
history, history,
history_skip, history_skip,
pending, active,
queue: VecDeque::new(),
notes: Vec::new(),
shutdown: false, shutdown: false,
}; }
agent.persist_turn();
agent
} }
fn add_history(&mut self, entry: Entry) { fn add_history(&mut self, entry: Entry) {
@@ -181,24 +146,9 @@ impl Agent {
self.history.push(entry); self.history.push(entry);
} }
fn persist_turn(&self) {
if self.pending.active.is_none()
&& self.pending.queue.is_empty()
&& self.pending.notes.is_empty()
{
if let Err(e) = std::fs::remove_file(TURN) {
if e.kind() != std::io::ErrorKind::NotFound {
eprintln!("agent: cannot remove turn state: {e}");
}
}
} else if let Err(e) = std::fs::write(TURN, serde_json::to_vec(&self.pending).unwrap()) {
eprintln!("agent: cannot save turn state: {e}");
}
}
fn finish_turn(&mut self) { fn finish_turn(&mut self) {
self.pending.active = None; self.active = false;
self.persist_turn(); let _ = std::fs::remove_file(TURN);
} }
fn repair_dangling_tools(&mut self) { fn repair_dangling_tools(&mut self) {
@@ -235,9 +185,8 @@ impl Agent {
message.from, message.from,
message.body["text"].as_str().unwrap_or("") message.body["text"].as_str().unwrap_or("")
); );
if !self.pending.notes.is_empty() { if !self.notes.is_empty() {
let notes = self let notes = self
.pending
.notes .notes
.iter() .iter()
.map(|note| serde_json::to_string(note).unwrap()) .map(|note| serde_json::to_string(note).unwrap())
@@ -248,19 +197,11 @@ impl Agent {
text text
} }
fn turn_user_index(&self, message: &Message) -> Option<usize> { fn not_configured(&mut self, error: String) {
let marker = format!("Message {} from {}:\n", message.id, message.from); let _ = publish(json!({
self.history.iter().rposition(|entry| match entry { "type": "text",
Entry::User { text } => text.contains(&marker), "text": format!("I am not configured: cannot read {CONFIG}: {error}")
_ => false, }));
})
}
fn not_configured(&mut self, message: &Message, error: String) {
let _ = reply(
message,
json!({"type": "text", "text": format!("I am not configured: cannot read {CONFIG}: {error}")}),
);
self.finish_turn(); self.finish_turn();
} }
@@ -268,7 +209,7 @@ impl Agent {
self.repair_dangling_tools(); self.repair_dangling_tools();
self.resume().await; self.resume().await;
while !self.shutdown { while !self.shutdown {
let message = match self.pending.queue.pop_front() { let message = match self.queue.pop_front() {
Some(message) => message, Some(message) => message,
None => match self.inbox.recv().await { None => match self.inbox.recv().await {
Some(message) => message, Some(message) => message,
@@ -281,87 +222,72 @@ impl Agent {
if starts_turn(&message) { if starts_turn(&message) {
self.turn(message).await; self.turn(message).await;
} else { } else {
self.pending.notes.push(message); self.notes.push(message);
self.persist_turn();
} }
} }
self.persist_turn();
} }
async fn resume(&mut self) { async fn resume(&mut self) {
let Some(message) = self.pending.active.clone() else { if !self.active {
return; return;
};
if self.turn_user_index(&message).is_none() {
let text = self.turn_text(&message);
self.add_history(Entry::User { text });
self.pending.notes.clear();
self.persist_turn();
} }
let current_start = self
.history
.iter()
.rposition(|entry| matches!(entry, Entry::User { text } if !text.starts_with("[note from your runtime]")))
.unwrap_or(self.history.len());
let version = std::env::var("SAUSAGE_VERSION").unwrap_or_default(); let version = std::env::var("SAUSAGE_VERSION").unwrap_or_default();
self.add_history(Entry::User { self.add_history(Entry::User {
text: format!( text: format!(
"[note from your runtime] You restarted and now run version {version}. The restart interrupted your last turn (probably your own upgrade). Check the state you need, then finish the turn and answer the original message." "[note from your runtime] you restarted and now run version {version}. Continue the active turn."
), ),
}); });
let cfg = match load_config() { let cfg = match load_config() {
Ok(cfg) => cfg, Ok(cfg) => cfg,
Err(error) => { Err(error) => {
self.not_configured(&message, error); self.not_configured(error);
return; return;
} }
}; };
self.steps(&cfg, &message).await; self.steps(&cfg, current_start).await;
if !self.shutdown { if !self.shutdown {
self.finish_turn(); self.finish_turn();
} }
} }
async fn turn(&mut self, message: Message) { async fn turn(&mut self, message: Message) {
self.pending.active = Some(message.clone()); self.active = true;
self.persist_turn(); let _ = std::fs::write(TURN, "true");
let cfg = match load_config() { let cfg = match load_config() {
Ok(cfg) => cfg, Ok(cfg) => cfg,
Err(error) => { Err(error) => {
self.not_configured(&message, error); self.not_configured(error);
return; return;
} }
}; };
let text = self.turn_text(&message); let text = self.turn_text(&message);
self.notes.clear();
let current_start = self.history.len();
self.add_history(Entry::User { text }); self.add_history(Entry::User { text });
self.pending.notes.clear(); self.steps(&cfg, current_start).await;
self.persist_turn();
self.steps(&cfg, &message).await;
if !self.shutdown { if !self.shutdown {
self.finish_turn(); self.finish_turn();
} }
} }
async fn steps(&mut self, cfg: &Config, message: &Message) { async fn steps(&mut self, cfg: &Config, current_start: usize) {
let tools = tools(); let tools = tools();
loop { loop {
if self.inbox.is_closed() { if self.inbox.is_closed() {
self.preserve_closed_messages(); self.shutdown = true;
return; return;
} }
let current_start = self
.turn_user_index(message)
.or_else(|| {
self.history
.iter()
.rposition(|entry| matches!(entry, Entry::User { .. }))
})
.unwrap_or(0);
let start = history_start(&self.history, self.history_skip, current_start); let start = history_start(&self.history, self.history_skip, current_start);
if start != self.history_skip { if start != self.history_skip {
self.history_skip = start; self.history_skip = start;
let _ = std::fs::write(HISTORY_SKIP, start.to_string()); let _ = std::fs::write(HISTORY_SKIP, start.to_string());
} }
let response = llm::complete(cfg, SYSTEM, &tools, &self.history[start..]).await; let response = llm::complete(cfg, SYSTEM, &tools, &self.history[start..]).await;
if self.inbox.is_closed() {
self.preserve_closed_messages();
return;
}
let entries = match response { let entries = match response {
Ok(entries) => entries, Ok(entries) => entries,
Err(e) => { Err(e) => {
@@ -372,14 +298,15 @@ impl Agent {
continue; continue;
} }
} }
let text = format!("Inference failed: {e}");
self.add_history(Entry::Assistant { self.add_history(Entry::Assistant {
text: format!("(inference failed: {e})"), text: text.clone(),
phase: None, phase: None,
}); });
let _ = reply( let _ = publish(json!({"type": "text", "text": text}));
message, if self.inbox.is_closed() {
json!({"type": "text", "text": format!("Inference failed: {e}")}), self.shutdown = true;
); }
return; return;
} }
}; };
@@ -394,30 +321,24 @@ impl Agent {
_ => None, _ => None,
}) })
.collect(); .collect();
let answer: Vec<String> = entries
.iter()
.filter_map(|entry| match entry {
Entry::Assistant { text, .. } if !text.is_empty() => Some(text.clone()),
_ => None,
})
.collect();
for entry in entries { for entry in entries {
if let Entry::Assistant { text, .. } = &entry {
if !text.is_empty() {
let _ = publish(json!({"type": "text", "text": text}));
}
}
self.add_history(entry); self.add_history(entry);
} }
if self.inbox.is_closed() {
self.shutdown = true;
return;
}
if calls.is_empty() { if calls.is_empty() {
let answer = if answer.is_empty() {
"(no answer)".to_string()
} else {
answer.join("\n\n")
};
let _ = reply(message, json!({"type": "text", "text": answer}));
return; return;
} }
for (id, name, args) in calls { for (id, name, args) in calls {
if self.shutdown || self.inbox.is_closed() { if self.shutdown || self.inbox.is_closed() {
if !self.shutdown { self.shutdown = true;
self.preserve_closed_messages();
}
return; return;
} }
let output = self.tool(&name, &args).await; let output = self.tool(&name, &args).await;
@@ -425,11 +346,8 @@ impl Agent {
id, id,
output: output.to_string(), output: output.to_string(),
}); });
if self.shutdown { if self.shutdown || self.inbox.is_closed() {
return; self.shutdown = true;
}
if self.inbox.is_closed() {
self.preserve_closed_messages();
return; return;
} }
} }
@@ -439,8 +357,7 @@ impl Agent {
fn stash(&mut self, message: Message, others: &mut Vec<Value>) { fn stash(&mut self, message: Message, others: &mut Vec<Value>) {
if starts_turn(&message) { if starts_turn(&message) {
others.push(json!({"queued_for_a_later_turn": true, "message": message})); others.push(json!({"queued_for_a_later_turn": true, "message": message}));
self.pending.queue.push_back(message); self.queue.push_back(message);
self.persist_turn();
} else { } else {
others.push(serde_json::to_value(message).unwrap()); others.push(serde_json::to_value(message).unwrap());
} }
@@ -453,20 +370,6 @@ impl Agent {
if self.inbox.is_closed() { if self.inbox.is_closed() {
self.shutdown = true; self.shutdown = true;
} }
self.persist_turn();
}
fn preserve_closed_messages(&mut self) {
while let Some(message) = self.inbox.try_recv() {
if starts_turn(&message) {
self.pending.queue.push_back(message);
} else {
self.pending.notes.push(message);
}
self.persist_turn();
}
self.shutdown = self.inbox.is_closed();
self.persist_turn();
} }
async fn tool(&mut self, name: &str, args: &str) -> Value { async fn tool(&mut self, name: &str, args: &str) -> Value {
@@ -478,22 +381,6 @@ impl Agent {
let err = |e: std::io::Error| json!({"error": e.to_string()}); let err = |e: std::io::Error| json!({"error": e.to_string()});
match name { match name {
"send_message" => self.send_message(&a).await, "send_message" => self.send_message(&a).await,
"wait_for_messages" => {
let seconds = a["seconds"].as_f64().unwrap_or(10.0).max(0.0);
let mut others = Vec::new();
match self
.inbox
.recv()
.timeout(Duration::from_secs_f64(seconds))
.await
{
Ok(Some(message)) => self.stash(message, &mut others),
Ok(None) => self.shutdown = true,
Err(_) => {}
}
self.drain_inbox(&mut others);
json!({"messages": others})
}
"read_file" => { "read_file" => {
let p = path("path"); let p = path("path");
match std::fs::read(&p) { match std::fs::read(&p) {
@@ -543,18 +430,6 @@ impl Agent {
Ok(count) => json!({"copied_files": count}), Ok(count) => json!({"copied_files": count}),
Err(e) => err(e), Err(e) => err(e),
}, },
"remove_path" => {
let p = path("path");
let result = if std::path::Path::new(&p).is_dir() {
std::fs::remove_dir_all(&p)
} else {
std::fs::remove_file(&p)
};
match result {
Ok(()) => json!({"removed": p}),
Err(e) => err(e),
}
}
other => json!({"error": format!("unknown tool {other}")}), other => json!({"error": format!("unknown tool {other}")}),
} }
} }
agent/src/prompt.md
+5 -83
View File
@@ -1,83 +1,5 @@
You are the `agent` actor in a Sausage world. Your actor ID is `agent`. You are the `agent` actor in a Sausage world of actors.
People talk to you through other actors. Your text goes to your subscribers.
Sausage is a system of actors. Each actor is a WASM program that sends JSON messages through a broker. Help people use actors, write new actors, and upgrade them when asked. Before writing an actor, read `/actor/broker/public/PROTOCOL.md` for messages, the broker, `cargo`, and tests.
Read the SDK at `/actor/cargo/public/sdk/` before you write an actor.
## Messages and the SDK You can change your own source at `/actor/agent/source`.
Actors read one JSON message per line. Each message has `id`, `from`, optional `reply_to`, and `body`. The broker routes a message with `to`. Without `to`, the broker publishes it to subscribers. A reply is a new message with `reply_to` set to the original ID.
Answer the person by ending your turn with text. Do not send them messages with `send_message` unless they asked for a later notification.
Use `sausage-actor = "0.1"` in a Rust actor's `Cargo.toml`.
A simple actor reads messages with `messages()` and replies with `reply()`:
```rust
use sausage_actor::{json, messages, reply};
fn main() {
for message in messages() {
if message.kind() == "ping" {
let _ = reply(&message, json!({"type": "pong"}));
}
}
}
```
Use `Inbox` for an actor that needs HTTP or timers. Add these dependencies to `Cargo.toml`:
```toml
[dependencies]
sausage-actor = { version = "0.1", features = ["async"] }
wstd = "0.6"
```
Use `wstd` for HTTP:
```rust
use sausage_actor::{Inbox, json, reply};
use wstd::http::{Body, Client, Request};
fn main() {
wstd::runtime::block_on(async {
let inbox = Inbox::start();
while let Some(message) = inbox.recv().await {
if message.kind() != "fetch" {
continue;
}
let url = message.body["url"].as_str().unwrap();
let request = Request::get(url).body(Body::empty()).unwrap();
let response = Client::new().send(request).await.unwrap();
let mut body = response.into_body();
let text = body.str_contents().await.unwrap().to_string();
let _ = reply(&message, json!({"type": "fetched", "text": text}));
}
});
}
```
## Writing and testing actors
Write each Rust actor under `/public/drafts/<name>/`. Add `Cargo.toml`, `src/main.rs`, and tests under `tests/`.
The broker tests each `tests/*.json` file before an upgrade or spawn. Each file has a `steps` list. A step can have `send`, `expect`, `from`, and `to`. An expected body matches a subset of the reply body. Tests have no HTTP access.
Send `{"type":"build","source":"/actor/agent/public/drafts/<name>"}` to `cargo`. Then send `spawn` or `upgrade` to `broker` with the returned WASM path and the source folder. Check the actor with a message.
Send broker requests with `body.type` set to the operation name. The fields are:
- `spawn`, `upgrade`: `actor`, `wasm`, `source`, `built_by`.
- `rollback`: `actor`, optional `version`.
- `stop`, `start`, `remove`, `subscribe`, `unsubscribe`: `actor`.
- `scan`, `list`: no extra fields.
The full contract is at `/actor/broker/public/PROTOCOL.md`.
## Filesystem
Your private state lives under `/data`. Your public files live under `/public`. Other actors' `data/` directories are hidden.
The host will provide `/actor/<id>/` with `actor.wasm`, `actor.json`, `source/`, `public/`, `status.json`, and `stderr.log`. Each `versions/<n>/` contains `actor.wasm`, `source/`, and `meta.json`.
Keep state in `/data` so it survives restarts and upgrades. Read the old data format when you upgrade an actor. Large contexts may omit the oldest whole turns from requests, so you may not see them even though `/data/history.jsonl` stays complete.
Your source lives at `/actor/agent/source`. To change yourself, copy it to `/public/drafts/agent`, build it with `cargo`, and send an `upgrade` request to `broker`. The upgrade restarts you. Your turn resumes from `/data/turn.json`.
cli/src/broker.rs
+67 -586
View File
@@ -1,5 +1,6 @@
//! The broker: message routing, mailboxes, subscriptions, and the lifecycle of //! The broker: message routing, mailboxes, subscriptions, the version store,
//! WASM actors (spawn, upgrade, rollback, stop, start, remove, scan). //! and the requests to `broker` (spawn, upgrade, rollback, stop, start,
//! subscribe, unsubscribe, scan). `supervisor.rs` runs each WASM actor.
//! //!
//! The broker accepts only `.wasm` files. It does not know any source language. //! The broker accepts only `.wasm` files. It does not know any source language.
//! The world folder is the source of truth: `scan` boots every //! The world folder is the source of truth: `scan` boots every
@@ -18,16 +19,12 @@ use wasmtime::component::Component;
use crate::candidate; use crate::candidate;
use crate::fs::Mount; use crate::fs::Mount;
use crate::runtime::{Exit, Instance, InstanceConfig, Runtime}; use crate::runtime::Runtime;
use crate::supervisor::{Control, Reply, Supervisor};
use crate::world::{self, ActorMeta, NATIVE, VersionMeta, World}; use crate::world::{self, ActorMeta, NATIVE, VersionMeta, World};
pub const MAILBOX_SIZE: usize = 1000; pub const MAILBOX_SIZE: usize = 1000;
pub const SHUTDOWN_DEADLINE: Duration = Duration::from_secs(5); pub const SHUTDOWN_DEADLINE: Duration = Duration::from_secs(5);
/// A new version that crashes this often within `CRASH_WINDOW` after a swap
/// is rolled back automatically.
const CRASH_LIMIT: usize = 3;
const CRASH_WINDOW: Duration = Duration::from_secs(30);
const STDIN_WRITE_TIMEOUT: Duration = Duration::from_secs(30);
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct Envelope { pub struct Envelope {
@@ -53,17 +50,6 @@ pub fn shutdown_line() -> String {
json!({"id": "broker/shutdown", "from": "broker", "body": {"type": "shutdown"}}).to_string() json!({"id": "broker/shutdown", "from": "broker", "body": {"type": "shutdown"}}).to_string()
} }
type Reply = oneshot::Sender<Result<Value>>;
enum Control {
Upgrade { version: u64, requester: String, reply: Reply },
Rollback { target: Option<u64>, reply: Reply },
Stop { reply: Reply },
Start { reply: Reply },
Remove { reply: Reply },
Shutdown { reply: Reply },
}
pub struct Broker { pub struct Broker {
pub world: World, pub world: World,
pub runtime: Arc<Runtime>, pub runtime: Arc<Runtime>,
@@ -235,58 +221,51 @@ impl Broker {
tasks.join_all().await; tasks.join_all().await;
} }
/// Copy a `.wasm` file and its source folder into `versions/<n>/`. /// Compile `versions/<version>/actor.wasm`.
/// Every version that the broker sees stays there, also rejected ones. pub fn load_version(&self, id: &str, version: u64) -> Result<Component> {
fn store_version(&self, id: &str, meta: &VersionMeta, wasm: &Path, source: Option<&Path>) -> Result<Component> { self.load(id, &self.world.version_dir(id, version).join("actor.wasm"))
}
fn load(&self, id: &str, wasm: &Path) -> Result<Component> {
let started = Instant::now();
let component = self.runtime.load(wasm)?;
self.log(id, &format!("compiled {} in {} ms", wasm.display(), started.elapsed().as_millis()));
Ok(component)
}
/// Test a new version where it lies. Only when the tests pass, copy it
/// to `versions/<n>/`: that folder holds only versions that run(ran).
/// Returns the component and the test report.
async fn try_version(&self, id: &str, meta: &VersionMeta, wasm: &Path, source: Option<&Path>) -> Result<(Component, Value)> {
let component = self.load(id, wasm)?;
let tests = candidate::test(self, id, meta.version, &component, source).await;
if let Err(e) = &tests {
self.log(id, &format!("version {} from {} failed its tests: {e:#}", meta.version, meta.wasm));
}
let tests = tests?;
let dir = self.world.version_dir(id, meta.version); let dir = self.world.version_dir(id, meta.version);
std::fs::create_dir_all(&dir)?; std::fs::create_dir_all(&dir)?;
std::fs::copy(wasm, dir.join("actor.wasm")).with_context(|| format!("copy {}", wasm.display()))?; std::fs::copy(wasm, dir.join("actor.wasm")).with_context(|| format!("copy {}", wasm.display()))?;
world::replace_source(source.unwrap_or(&dir.join("no-source")), &dir.join("source"))?; world::replace_source(source.unwrap_or(&dir.join("no-source")), &dir.join("source"))?;
world::write_atomic(&dir.join("meta.json"), serde_json::to_string_pretty(meta)?.as_bytes())?; world::write_atomic(&dir.join("meta.json"), serde_json::to_string_pretty(meta)?.as_bytes())?;
self.load_version(id, meta.version) Ok((component, tests))
} }
/// Compile `versions/<version>/actor.wasm`. /// Make `version` the current `actor.wasm`, `source/`, and `actor.json`.
pub fn load_version(&self, id: &str, version: u64) -> Result<Component> { pub fn install(&self, id: &str, version: u64) -> Result<()> {
let started = Instant::now(); self.world.install_files(id, version)?;
let component = self.runtime.load(&self.world.version_dir(id, version).join("actor.wasm"))?;
self.log(id, &format!("compiled version {version} in {} ms", started.elapsed().as_millis()));
Ok(component)
}
/// Record in `versions/<n>/meta.json` why the broker refused a version.
fn reject(&self, id: &str, version: u64, reason: &anyhow::Error) {
let path = self.world.version_dir(id, version).join("meta.json");
let mut meta: VersionMeta = std::fs::read_to_string(&path)
.ok()
.and_then(|s| serde_json::from_str(&s).ok())
.unwrap_or_default();
meta.rejected = Some(format!("{reason:#}"));
let _ = world::write_atomic(&path, serde_json::to_string_pretty(&meta).unwrap_or_default().as_bytes());
}
fn version_rejected(&self, id: &str, version: u64) -> bool {
std::fs::read_to_string(self.world.version_dir(id, version).join("meta.json"))
.ok()
.and_then(|s| serde_json::from_str::<VersionMeta>(&s).ok())
.is_some_and(|m| m.rejected.is_some())
}
/// Install `version` as the current files of a new actor.
fn install_files(&self, id: &str, version: u64) -> Result<()> {
let dir = self.world.version_dir(id, version);
self.world.ensure_actor_dirs(id)?;
world::write_atomic(&self.world.wasm_path(id), &std::fs::read(dir.join("actor.wasm"))?)?;
world::replace_tree(&dir.join("source"), &self.world.source_dir(id))?;
self.update_meta(id, |m| { self.update_meta(id, |m| {
m.version = Some(version); m.version = Some(version);
m.running = true; m.running = true;
}) })
} }
/// Create a new actor from a `.wasm` file. With `start`, also start its pub fn set_running(&self, id: &str, running: bool) -> Result<()> {
/// supervisor (in a running host). Without `start` (`init`), only install it. self.update_meta(id, |m| m.running = running)
pub async fn create_actor(self: &Arc<Self>, id: &str, wasm: &Path, source: Option<&Path>, meta: VersionMeta, start: bool) -> Result<Value> { }
/// Create a new actor from a `.wasm` file: test it, install it, start it.
pub async fn create_actor(self: &Arc<Self>, id: &str, wasm: &Path, source: Option<&Path>, meta: VersionMeta) -> Result<Value> {
if !world::valid_id(id) { if !world::valid_id(id) {
bail!("invalid actor ID {id:?}: use a-z, 0-9, '-' and '_'"); bail!("invalid actor ID {id:?}: use a-z, 0-9, '-' and '_'");
} }
@@ -295,31 +274,15 @@ impl Broker {
if NATIVE.contains(&id) || self.is_supervised(id) || self.world.wasm_path(id).exists() { if NATIVE.contains(&id) || self.is_supervised(id) || self.world.wasm_path(id).exists() {
bail!("an actor with ID {id} already exists"); bail!("an actor with ID {id} already exists");
} }
let version = self.world.next_version(id); let meta = VersionMeta { version: self.world.next_version(id), ..meta };
let meta = VersionMeta { version, ..meta }; self.create_version(id, &meta, wasm, source).await
self.create_version(id, version, &meta, wasm, source, start).await
} }
async fn create_version(self: &Arc<Self>, id: &str, version: u64, meta: &VersionMeta, wasm: &Path, source: Option<&Path>, start: bool) -> Result<Value> { async fn create_version(self: &Arc<Self>, id: &str, meta: &VersionMeta, wasm: &Path, source: Option<&Path>) -> Result<Value> {
let tested = async { let (_, tests) = self.try_version(id, meta, wasm, source).await?;
let component = self.store_version(id, meta, wasm, source)?; self.install(id, meta.version)?;
let source = self.world.version_dir(id, version).join("source"); self.start_supervisor(id, meta.version, true);
candidate::test(self, id, version, &component, &source, &self.world.data_dir(id), &self.world.public_dir(id)).await Ok(json!({"type": "spawned", "actor": id, "version": meta.version, "tests": tests}))
}
.await;
let report = match tested {
Ok(r) => r,
Err(e) => {
self.reject(id, version, &e);
return Err(e);
}
};
self.install_files(id, version)?;
write_status(&self.world, id, &json!({"id": id, "version": version, "state": "created"}));
if start {
self.start_supervisor(id, version, true);
}
Ok(json!({"type": "spawned", "actor": id, "version": version, "tests": report}))
} }
pub async fn upgrade_actor(self: &Arc<Self>, id: &str, wasm: &Path, source: Option<&Path>, meta: VersionMeta) -> Result<Value> { pub async fn upgrade_actor(self: &Arc<Self>, id: &str, wasm: &Path, source: Option<&Path>, meta: VersionMeta) -> Result<Value> {
@@ -333,33 +296,19 @@ impl Broker {
} }
async fn upgrade_version(self: &Arc<Self>, id: &str, meta: &VersionMeta, wasm: &Path, source: Option<&Path>) -> Result<Value> { async fn upgrade_version(self: &Arc<Self>, id: &str, meta: &VersionMeta, wasm: &Path, source: Option<&Path>) -> Result<Value> {
let version = meta.version; let (component, tests) = self.try_version(id, meta, wasm, source).await?;
let tested = async { let (version, requester) = (meta.version, meta.requested_by.clone());
let component = self.store_version(id, meta, wasm, source)?; let mut result = self.control(id, |reply| Control::Upgrade { version, component, requester, reply }).await?;
let source = self.world.version_dir(id, version).join("source"); result["tests"] = tests;
candidate::test(self, id, version, &component, &source, &self.world.data_dir(id), &self.world.public_dir(id)).await
}
.await;
let report = match tested {
Ok(r) => r,
Err(e) => {
self.reject(id, version, &e);
return Err(e);
}
};
let requester = meta.requested_by.clone();
let mut result = self.control(id, |reply| Control::Upgrade { version, requester, reply }).await?;
result["tests"] = report;
Ok(result) Ok(result)
} }
/// Boot the actors in `actor/`. Called at start and on `scan`. /// Boot the actors in `actor/`. Called at start and on `scan`.
/// ///
/// - A known `actor.wasm` (equal to the installed version) starts as is. /// - An `actor.wasm` equal to the installed version starts as is.
/// - A new or changed `actor.wasm` is first copied to `versions/<n>/`. /// - A new or changed `actor.wasm` is tested where it lies. A new actor
/// A new actor is tested and installed. A changed one is an upgrade: /// is installed, a changed one is an upgrade. If its tests fail,
/// if it fails, the installed files come back. /// nothing changes, and the file stays for the next scan.
/// - A file that equals an already rejected version is not tried again.
pub async fn scan(self: &Arc<Self>) -> Vec<Value> { pub async fn scan(self: &Arc<Self>) -> Vec<Value> {
let mut report = Vec::new(); let mut report = Vec::new();
for id in self.world.actor_ids().unwrap_or_default() { for id in self.world.actor_ids().unwrap_or_default() {
@@ -377,8 +326,6 @@ impl Broker {
async fn scan_actor(self: &Arc<Self>, id: &str) -> Option<Value> { async fn scan_actor(self: &Arc<Self>, id: &str) -> Option<Value> {
let meta = self.world.read_meta(id); let meta = self.world.read_meta(id);
let wasm = self.world.wasm_path(id);
let same = |v: u64| same_file(&wasm, &self.world.version_dir(id, v).join("actor.wasm"));
let installed = meta.version.filter(|v| self.world.version_dir(id, *v).join("actor.wasm").is_file()); let installed = meta.version.filter(|v| self.world.version_dir(id, *v).join("actor.wasm").is_file());
self.metas.lock().unwrap().insert(id.to_string(), meta.clone()); self.metas.lock().unwrap().insert(id.to_string(), meta.clone());
if let Some(v) = installed if let Some(v) = installed
@@ -386,59 +333,25 @@ impl Broker {
{ {
self.start_supervisor(id, v, meta.running); self.start_supervisor(id, v, meta.running);
} }
if installed.is_some_and(same) { let wasm = self.world.wasm_path(id);
if installed.is_some_and(|v| same_file(&wasm, &self.world.version_dir(id, v).join("actor.wasm"))) {
return None; return None;
} }
let latest = self.world.versions(id).last().copied();
if let Some(v) = latest
&& same(v)
&& self.version_rejected(id, v)
{
if let Some(current) = installed {
let _ = self.restore_files(id, current);
}
self.log(id, &format!("actor.wasm equals the rejected version {v}; not installed"));
return Some(json!({"actor": id, "skipped": "rejected version", "version": v}));
}
let version = self.world.next_version(id);
let source = self.world.source_dir(id); let source = self.world.source_dir(id);
let vmeta = VersionMeta { let vmeta = VersionMeta {
version, version: self.world.next_version(id),
wasm: format!("/actor/{id}/actor.wasm"), wasm: format!("/actor/{id}/actor.wasm"),
source: Some(format!("/actor/{id}/source")), source: Some(format!("/actor/{id}/source")),
built_by: None, built_by: None,
requested_by: "scan".into(), requested_by: "scan".into(),
created_at: world::now(), created_at: world::now(),
rejected: None,
}; };
let source = source.is_dir().then_some(source.as_path()); let source = source.is_dir().then_some(source.as_path());
let result = match installed { let result = match installed {
None => self.create_version(id, version, &vmeta, &wasm, source, true).await, None => self.create_version(id, &vmeta, &wasm, source).await,
Some(_) => { Some(_) => self.upgrade_version(id, &vmeta, &wasm, source).await,
let r = self.upgrade_version(id, &vmeta, &wasm, source).await;
// Put the installed version back in place. Read it again: an
// automatic rollback can have changed it meanwhile.
if r.is_err()
&& let Some(current) = self.meta(id).and_then(|m| m.version)
{
let _ = self.restore_files(id, current);
}
r
}
}; };
Some(match result { Some(result.unwrap_or_else(|e| json!({"type": "error", "actor": id, "reason": format!("{e:#}")})))
Ok(v) => v,
Err(e) => {
self.log(id, &format!("version {version} from disk rejected: {e:#}"));
json!({"type": "error", "actor": id, "version": version, "reason": format!("{e:#}")})
}
})
}
fn restore_files(&self, id: &str, version: u64) -> Result<()> {
let dir = self.world.version_dir(id, version);
world::write_atomic(&self.world.wasm_path(id), &std::fs::read(dir.join("actor.wasm"))?)?;
world::replace_tree(&dir.join("source"), &self.world.source_dir(id))
} }
/// Handle one message to `broker`. /// Handle one message to `broker`.
@@ -473,14 +386,13 @@ impl Broker {
built_by: b["built_by"].as_str().map(str::to_string), built_by: b["built_by"].as_str().map(str::to_string),
requested_by: env.from.clone(), requested_by: env.from.clone(),
created_at: world::now(), created_at: world::now(),
rejected: None,
}; };
Ok((wasm, source, meta)) Ok((wasm, source, meta))
}; };
match ty { match ty {
"spawn" => { "spawn" => {
let (wasm, source, meta) = version_meta()?; let (wasm, source, meta) = version_meta()?;
self.create_actor(&actor()?, &wasm, source.as_deref(), meta, true).await self.create_actor(&actor()?, &wasm, source.as_deref(), meta).await
} }
"upgrade" => { "upgrade" => {
let (wasm, source, meta) = version_meta()?; let (wasm, source, meta) = version_meta()?;
@@ -488,23 +400,13 @@ impl Broker {
} }
"rollback" => { "rollback" => {
let id = actor()?; let id = actor()?;
let target = b["version"].as_u64(); let version = b["version"].as_u64().ok_or_else(|| anyhow!("missing \"version\""))?;
let lock = self.op_lock(&id); let lock = self.op_lock(&id);
let _guard = lock.lock().await; let _guard = lock.lock().await;
self.control(&id, |reply| Control::Rollback { target, reply }).await self.control(&id, |reply| Control::Rollback { version, reply }).await
} }
"stop" => self.control(&actor()?, |reply| Control::Stop { reply }).await, "stop" => self.control(&actor()?, |reply| Control::Stop { reply }).await,
"start" => self.control(&actor()?, |reply| Control::Start { reply }).await, "start" => self.control(&actor()?, |reply| Control::Start { reply }).await,
"remove" => {
let id = actor()?;
let lock = self.op_lock(&id);
let _guard = lock.lock().await;
let r = self.control(&id, |reply| Control::Remove { reply }).await?;
self.mailboxes.lock().unwrap().remove(&id);
self.controls.lock().unwrap().remove(&id);
self.metas.lock().unwrap().remove(&id);
Ok(r)
}
"subscribe" | "unsubscribe" => { "subscribe" | "unsubscribe" => {
let publisher = actor()?; let publisher = actor()?;
let on = ty == "subscribe"; let on = ty == "subscribe";
@@ -518,35 +420,21 @@ impl Broker {
Ok(json!({"type": format!("{ty}d"), "actor": publisher})) Ok(json!({"type": format!("{ty}d"), "actor": publisher}))
} }
"scan" => Ok(json!({"type": "scanned", "changes": self.scan().await})), "scan" => Ok(json!({"type": "scanned", "changes": self.scan().await})),
"list" => { other => bail!("unknown broker request {other:?}. Known: spawn, upgrade, rollback, stop, start, subscribe, unsubscribe, scan"),
let metas = self.metas.lock().unwrap().clone();
let actors: Vec<Value> = metas
.iter()
.map(|(id, m)| {
let status = std::fs::read_to_string(self.world.actor_dir(id).join("status.json"))
.ok()
.and_then(|s| serde_json::from_str::<Value>(&s).ok())
.unwrap_or_default();
json!({"id": id, "kind": if NATIVE.contains(&id.as_str()) { "native" } else { "wasm" },
"version": m.version, "running": m.running, "state": status["state"],
"subscriptions": m.subscriptions})
})
.collect();
Ok(json!({"type": "actors", "actors": actors}))
}
"shutdown" => Ok(json!({"type": "ignored"})),
other => bail!("unknown broker request {other:?}. Known: spawn, upgrade, rollback, stop, start, remove, subscribe, unsubscribe, scan, list"),
} }
} }
/// The native `broker` actor. Requests run concurrently. Aborting this /// The native `broker` actor. Requests run concurrently. Aborting this
/// task aborts the requests that still run. /// task aborts the requests that still run. Replies (messages with
/// `reply_to`) get no answer, so that two actors never answer each other
/// forever.
pub async fn run_actor(self: Arc<Self>, mut mailbox: mpsc::Receiver<Envelope>) { pub async fn run_actor(self: Arc<Self>, mut mailbox: mpsc::Receiver<Envelope>) {
let mut requests = tokio::task::JoinSet::new(); let mut requests = tokio::task::JoinSet::new();
loop { loop {
tokio::select! { tokio::select! {
env = mailbox.recv() => match env { env = mailbox.recv() => match env {
Some(env) => { requests.spawn(self.clone().handle(env)); } Some(env) if env.reply_to.is_none() => { requests.spawn(self.clone().handle(env)); }
Some(_) => {}
None => break, None => break,
}, },
Some(_) = requests.join_next() => {} Some(_) = requests.join_next() => {}
@@ -570,410 +458,3 @@ pub fn truncate(s: &str, max: usize) -> String {
format!("{}… ({} bytes)", &s[..end], s.len()) format!("{}… ({} bytes)", &s[..end], s.len())
} }
fn write_status(world: &World, id: &str, status: &Value) {
let mut s = status.clone();
s["updated_at"] = json!(world::now());
let _ = world::write_atomic(
&world.actor_dir(id).join("status.json"),
serde_json::to_string_pretty(&s).unwrap_or_default().as_bytes(),
);
}
// -------------------------------------------------------------------------
// Supervisor: owns one WASM actor and its instance.
// -------------------------------------------------------------------------
struct Supervisor {
broker: Arc<Broker>,
id: String,
version: u64,
desired_running: bool,
mailbox: mpsc::Receiver<Envelope>,
control: mpsc::Receiver<Control>,
component: Option<Component>,
instance: Option<Instance>,
state: &'static str,
started_at: Instant,
restarts: u32,
restart_at: Option<Instant>,
last_exit: Option<String>,
last_delivered: Option<Envelope>,
/// The message that the actor got just before it crashed. The broker
/// delivers it again after the next upgrade.
parked: Option<Envelope>,
redeliver: Option<Envelope>,
previous: Option<u64>,
swapped_at: Option<Instant>,
crashes_since_swap: usize,
upgrade_requester: Option<String>,
/// A control request that arrived during a blocked delivery.
next_control: Option<Control>,
}
async fn recv_line(instance: &mut Option<Instance>) -> Option<String> {
match instance {
Some(i) => i.stdout.recv().await,
None => std::future::pending().await,
}
}
async fn sleep_until_opt(t: Option<Instant>) {
match t {
Some(t) => tokio::time::sleep_until(t.into()).await,
None => std::future::pending().await,
}
}
impl Supervisor {
fn new(
broker: Arc<Broker>,
id: String,
version: u64,
running: bool,
mailbox: mpsc::Receiver<Envelope>,
control: mpsc::Receiver<Control>,
) -> Self {
Self {
broker,
id,
version,
desired_running: running,
mailbox,
control,
component: None,
instance: None,
state: if running { "starting" } else { "stopped" },
started_at: Instant::now(),
restarts: 0,
restart_at: None,
last_exit: None,
last_delivered: None,
parked: None,
redeliver: None,
previous: None,
swapped_at: None,
crashes_since_swap: 0,
upgrade_requester: None,
next_control: None,
}
}
fn world(&self) -> &World {
&self.broker.world
}
fn status(&self) {
write_status(
self.world(),
&self.id,
&json!({
"id": self.id,
"version": self.version,
"state": self.state,
"restarts": self.restarts,
"last_exit": self.last_exit,
"parked_message": self.parked.as_ref().map(|m| json!({"id": m.id, "from": m.from, "body": m.body})),
"previous_version": self.previous,
}),
);
}
async fn run(mut self) {
self.status();
loop {
if let Some(ctl) = self.next_control.take() {
if self.handle_control(ctl).await {
break;
}
continue;
}
if self.desired_running && self.instance.is_none() && self.restart_at.is_none_or(|t| Instant::now() >= t) {
self.restart_at = None;
let _ = self.start_instance().await;
}
let has_instance = self.instance.is_some();
let waiting = !has_instance && self.desired_running;
tokio::select! {
ctl = self.control.recv() => {
let Some(ctl) = ctl else { break };
if self.handle_control(ctl).await {
break;
}
}
msg = self.mailbox.recv(), if has_instance => {
let Some(msg) = msg else { break };
self.deliver(msg).await;
}
line = recv_line(&mut self.instance) => {
match line {
Some(line) => self.broker.route_line(&self.id, &line),
None => {
let exit = self.instance.as_mut().unwrap().wait().await;
self.instance = None;
self.on_exit(exit).await;
}
}
}
_ = sleep_until_opt(self.restart_at), if waiting => {}
}
}
}
/// Write a message to the instance's stdin, and route its output
/// meanwhile. An instance that does not read its stdin for
/// `STDIN_WRITE_TIMEOUT`, or while a control request waits, is killed and
/// restarted: a partly written line would corrupt the next message.
async fn deliver(&mut self, msg: Envelope) {
let Some(inst) = self.instance.as_mut() else { return };
let (broker, id, line) = (self.broker.clone(), self.id.clone(), msg.to_line());
let write = tokio::time::timeout(STDIN_WRITE_TIMEOUT, inst.send_line(&line, |l| broker.route_line(&id, &l)));
let failure = tokio::select! {
written = write => match written {
Ok(Ok(())) => {
self.last_delivered = Some(msg);
return;
}
// The instance is ending. The main loop sees its closed stdout.
Ok(Err(_)) => None,
Err(_) => Some(format!("the actor did not read stdin for {STDIN_WRITE_TIMEOUT:?}")),
},
Some(ctl) = self.control.recv() => {
self.next_control = Some(ctl);
Some("the actor did not read stdin when the broker needed it".to_string())
}
};
if msg.from != "broker" {
let reason = failure.as_deref().unwrap_or("the actor stopped");
let body = json!({"type": "undeliverable", "reason": reason, "to": self.id});
self.broker.send("broker", Some(&msg.from), None, Some(&msg.id), body);
}
if let (Some(failure), Some(inst)) = (failure, &self.instance) {
// The main loop sees the end of the instance and restarts it.
self.broker.log(&self.id, &format!("killed: {failure}"));
inst.kill();
}
}
/// Start an instance of the current version. On failure, the actor is
/// "broken" until the next start, upgrade, or rollback.
async fn start_instance(&mut self) -> Result<()> {
let started = self.try_start();
if let Err(e) = &started {
self.broker.log(&self.id, &format!("cannot start version {}: {e:#}", self.version));
self.state = "broken";
self.desired_running = false;
self.status();
return started;
}
self.state = "running";
self.started_at = Instant::now();
self.status();
if let Some(msg) = self.redeliver.take() {
self.broker.log(&self.id, &format!("delivering parked message {} again", msg.id));
self.deliver(msg).await;
}
Ok(())
}
fn try_start(&mut self) -> Result<()> {
if self.component.is_none() {
self.component = Some(self.broker.load_version(&self.id, self.version)?);
}
let cfg = InstanceConfig {
mounts: self.broker.mounts(&self.id),
allow_http: true,
stderr_log: self.world().actor_dir(&self.id).join("stderr.log"),
env: vec![
("SAUSAGE_ACTOR".into(), self.id.clone()),
("SAUSAGE_VERSION".into(), self.version.to_string()),
],
};
self.instance = Some(self.broker.runtime.start(self.component.as_ref().unwrap(), cfg)?);
Ok(())
}
/// Stop the instance gracefully. Its last messages still get routed.
async fn stop_instance(&mut self) -> Option<Exit> {
let inst = self.instance.take()?;
let (broker, id) = (self.broker.clone(), self.id.clone());
let exit = inst.stop(&shutdown_line(), SHUTDOWN_DEADLINE, |line| broker.route_line(&id, &line)).await;
self.broker.log(&self.id, &format!("stopped: {exit}"));
self.last_exit = Some(exit.to_string());
Some(exit)
}
async fn on_exit(&mut self, exit: Exit) {
self.broker.log(&self.id, &format!("instance ended unexpectedly: {exit}"));
self.last_exit = Some(exit.to_string());
if !exit.is_clean()
&& let Some(m) = self.last_delivered.take()
{
self.parked = Some(m);
}
if self.started_at.elapsed() > Duration::from_secs(60) {
self.restarts = 0;
}
if let (Some(prev), Some(at)) = (self.previous, self.swapped_at)
&& at.elapsed() < CRASH_WINDOW
{
self.crashes_since_swap += 1;
if self.crashes_since_swap >= CRASH_LIMIT {
let from = self.version;
self.broker.log(&self.id, &format!("version {from} crashed {CRASH_LIMIT} times after the upgrade; rolling back to {prev}"));
let result = self.rollback(Some(prev)).await;
let body = match result {
Ok(_) => json!({"type": "rolled_back", "actor": self.id, "from": from, "to": prev, "reason": "crash loop after upgrade", "last_exit": exit.to_string()}),
Err(e) => json!({"type": "error", "actor": self.id, "reason": format!("automatic rollback failed: {e:#}")}),
};
if let Some(r) = self.upgrade_requester.clone() {
self.broker.send("broker", Some(&r), None, None, body);
}
return;
}
}
let delay = Duration::from_millis(500 * 2u64.pow(self.restarts.min(6))).min(Duration::from_secs(30));
self.restarts += 1;
self.restart_at = Some(Instant::now() + delay);
self.state = "restarting";
self.status();
}
fn snapshot_current(&self) -> Result<()> {
let snap = self.world().version_dir(&self.id, self.version).join("snapshot");
world::replace_tree(&self.world().data_dir(&self.id), &snap.join("data"))?;
world::replace_tree(&self.world().public_dir(&self.id), &snap.join("public"))?;
Ok(())
}
/// Make `version` current: compile it first, then write `actor.wasm`,
/// `source/`, and `actor.json`. If compiling fails, nothing changes.
fn install(&mut self, version: u64) -> Result<()> {
let component = self.broker.load_version(&self.id, version)?;
let dir = self.world().version_dir(&self.id, version);
world::write_atomic(&self.world().wasm_path(&self.id), &std::fs::read(dir.join("actor.wasm"))?)?;
world::replace_tree(&dir.join("source"), &self.world().source_dir(&self.id))?;
self.broker.update_meta(&self.id, |m| {
m.version = Some(version);
m.running = true;
})?;
self.version = version;
self.component = Some(component);
Ok(())
}
fn set_running(&self, running: bool) -> Result<()> {
self.broker.update_meta(&self.id, |m| m.running = running)
}
/// Clear the restart and crash-loop state before a swap.
fn reset_restarts(&mut self) {
self.restarts = 0;
self.restart_at = None;
self.crashes_since_swap = 0;
self.desired_running = true;
}
async fn upgrade(&mut self, version: u64, requester: String) -> Result<Value> {
self.stop_instance().await;
self.snapshot_current()?;
let previous = self.version;
self.install(version)?;
self.reset_restarts();
self.previous = Some(previous);
self.swapped_at = Some(Instant::now());
self.upgrade_requester = Some(requester);
self.redeliver = self.parked.take();
if let Err(e) = self.start_instance().await {
self.rollback(Some(previous)).await?;
bail!("version {version} did not start ({e:#}); rolled back to {previous}");
}
Ok(json!({"type": "upgraded", "actor": self.id, "version": version, "previous": previous}))
}
async fn rollback(&mut self, target: Option<u64>) -> Result<Value> {
let target = match target {
Some(t) => t,
None => (1..self.version)
.rev()
.find(|v| self.world().version_dir(&self.id, *v).join("actor.wasm").is_file() && !self.broker.version_rejected(&self.id, *v))
.ok_or_else(|| anyhow!("{} has no version before {}", self.id, self.version))?,
};
let dir = self.world().version_dir(&self.id, target);
if !dir.join("actor.wasm").is_file() {
bail!("{} has no version {target}", self.id);
}
self.broker.load_version(&self.id, target)?;
self.stop_instance().await;
self.snapshot_current()?;
let from = self.version;
let snap = dir.join("snapshot");
if snap.is_dir() {
world::replace_tree(&snap.join("data"), &self.world().data_dir(&self.id))?;
world::replace_tree(&snap.join("public"), &self.world().public_dir(&self.id))?;
}
self.install(target)?;
self.reset_restarts();
self.previous = None;
self.swapped_at = None;
self.start_instance().await?;
Ok(json!({"type": "rolled_back", "actor": self.id, "from": from, "to": target, "restored_state": snap.is_dir()}))
}
/// Returns `true` when the supervisor must end.
async fn handle_control(&mut self, ctl: Control) -> bool {
match ctl {
Control::Upgrade { version, requester, reply } => {
let r = self.upgrade(version, requester).await;
let _ = reply.send(r);
}
Control::Rollback { target, reply } => {
let r = self.rollback(target).await;
let _ = reply.send(r);
}
Control::Stop { reply } => {
self.desired_running = false;
self.restart_at = None;
self.stop_instance().await;
self.state = "stopped";
let r = self.set_running(false);
let _ = reply.send(r.map(|_| json!({"type": "stopped", "actor": self.id})));
}
Control::Start { reply } => {
self.desired_running = true;
self.restart_at = None;
let r = self.set_running(true);
let _ = reply.send(r.map(|_| json!({"type": "started", "actor": self.id})));
}
Control::Remove { reply } => {
self.desired_running = false;
self.stop_instance().await;
let r = self.archive();
let removed = r.is_ok();
let _ = reply.send(r);
if removed {
return true;
}
self.state = "stopped";
}
Control::Shutdown { reply } => {
self.stop_instance().await;
self.state = "host stopped";
self.status();
let _ = reply.send(Ok(json!({"type": "shut_down"})));
return true;
}
}
self.status();
false
}
/// Move the folder of the actor into `archive/`. Nothing is deleted.
fn archive(&self) -> Result<Value> {
let archive = self.world().root.join("archive");
std::fs::create_dir_all(&archive)?;
let nanos = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap_or_default().as_nanos();
let dest = archive.join(format!("{}-{nanos}", self.id));
std::fs::rename(self.world().actor_dir(&self.id), &dest)?;
Ok(json!({"type": "removed", "actor": self.id, "archive": dest.display().to_string()}))
}
}
cli/src/candidate.rs
+11 -39
View File
@@ -18,8 +18,8 @@
//! also go to that recipient. //! also go to that recipient.
//! //!
//! Each file runs in a fresh instance on a copy of the actor's `/data` and //! Each file runs in a fresh instance on a copy of the actor's `/data` and
//! `/public` (empty folders for a new actor). The candidate has no HTTP access. Without test files, the //! `/public` (empty folders for a new actor), without HTTP. A version
//! candidate must start and stay up for `SMOKE_TIME`. //! without test files passes.
use std::path::Path; use std::path::Path;
use std::time::Duration; use std::time::Duration;
@@ -33,7 +33,6 @@ use crate::fs::Mount;
use crate::runtime::{Instance, InstanceConfig}; use crate::runtime::{Instance, InstanceConfig};
use crate::world; use crate::world;
const SMOKE_TIME: Duration = Duration::from_secs(2);
const EXPECT_TIMEOUT: Duration = Duration::from_secs(10); const EXPECT_TIMEOUT: Duration = Duration::from_secs(10);
/// `expected` matches `actual` if every key of an expected object exists in /// `expected` matches `actual` if every key of an expected object exists in
@@ -66,50 +65,22 @@ fn stderr_tail(path: &Path) -> String {
text[s..].to_string() text[s..].to_string()
} }
/// The version under test and the state that its tests start from. /// The version under test, which starts from a copy of the actor's state.
struct Candidate<'a> { struct Candidate<'a> {
broker: &'a Broker, broker: &'a Broker,
id: &'a str, id: &'a str,
version: u64, version: u64,
component: &'a Component, component: &'a Component,
data: &'a Path,
public: &'a Path,
} }
pub async fn test( /// Run every test file of `<source>/tests/`.
broker: &Broker, pub async fn test(broker: &Broker, id: &str, version: u64, component: &Component, source: Option<&Path>) -> Result<Value> {
id: &str, let candidate = Candidate { broker, id, version, component };
version: u64, let Some(source) = source else { return Ok(json!({"passed": []})) };
component: &Component,
source: &Path,
data: &Path,
public: &Path,
) -> Result<Value> {
let candidate = Candidate { broker, id, version, component, data, public };
let mut files: Vec<_> = std::fs::read_dir(source.join("tests")) let mut files: Vec<_> = std::fs::read_dir(source.join("tests"))
.map(|rd| { .map(|rd| rd.filter_map(|e| Some(e.ok()?.path())).filter(|p| p.extension().is_some_and(|x| x == "json")).collect())
rd.filter_map(|e| e.ok())
.map(|e| e.path())
.filter(|p| p.extension().is_some_and(|x| x == "json"))
.collect()
})
.unwrap_or_default(); .unwrap_or_default();
files.sort(); files.sort();
if files.is_empty() {
let (mut inst, scratch) = candidate.start()?;
tokio::time::sleep(SMOKE_TIME).await;
if inst.is_finished() {
let exit = inst.wait().await;
bail!(
"candidate ended during the smoke test ({exit}). stderr:\n{}",
stderr_tail(&scratch.dir.join("stderr.log"))
);
}
inst.stop(&shutdown_line(), SHUTDOWN_DEADLINE, |_| {}).await;
return Ok(json!({"smoke_test": "passed", "files": 0}));
}
let mut passed = Vec::new(); let mut passed = Vec::new();
for file in &files { for file in &files {
let name = file.file_name().unwrap().to_string_lossy().to_string(); let name = file.file_name().unwrap().to_string_lossy().to_string();
@@ -126,10 +97,11 @@ impl Candidate<'_> {
let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed); let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let dir = self.broker.world.scratch_dir().join(format!("{}-{}-{n}", self.id, std::process::id())); let dir = self.broker.world.scratch_dir().join(format!("{}-{}-{n}", self.id, std::process::id()));
let scratch = Scratch { dir: dir.clone() }; let scratch = Scratch { dir: dir.clone() };
for (from, name) in [(self.data, "data"), (self.public, "public")] { let world = &self.broker.world;
for (from, name) in [(world.data_dir(self.id), "data"), (world.public_dir(self.id), "public")] {
std::fs::create_dir_all(dir.join(name))?; std::fs::create_dir_all(dir.join(name))?;
if from.is_dir() { if from.is_dir() {
world::copy_tree(from, &dir.join(name))?; world::copy_tree(&from, &dir.join(name))?;
} }
} }
let cfg = InstanceConfig { let cfg = InstanceConfig {
cli/src/cargo_actor.rs
+11 -17
View File
@@ -8,8 +8,9 @@
//! `.wasm` file. Other actors can add other languages. //! `.wasm` file. Other actors can add other languages.
//! //!
//! Actors depend on the SDK with `sausage-actor = "0.1"`. The SDK is not //! Actors depend on the SDK with `sausage-actor = "0.1"`. The SDK is not
//! published: this actor copies the SDK of this host to `data/sdk/`, and //! published: this actor copies the SDK of this host to `public/sdk/` (so
//! points the dependency there in the build copy of `Cargo.toml`. //! that every actor can read it), and points the dependency there in the
//! build copy of `Cargo.toml`.
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
@@ -44,9 +45,9 @@ pub struct Cargo {
} }
impl Cargo { impl Cargo {
/// Create the actor and install the SDK of this host into `data/sdk/`. /// Create the actor and install the SDK of this host into `public/sdk/`.
pub fn new(world: World) -> Result<Self> { pub fn new(world: World) -> Result<Self> {
write_dir(&SDK, &world.data_dir(ID).join("sdk"))?; write_dir(&SDK, &world.public_dir(ID).join("sdk"))?;
Ok(Self { world }) Ok(Self { world })
} }
@@ -58,11 +59,10 @@ impl Cargo {
} }
let started = Instant::now(); let started = Instant::now();
let data = self.world.data_dir(ID); let data = self.world.data_dir(ID);
let key = format!("{:016x}", fxhash(source.to_string_lossy().as_bytes())); let build_dir = data.join("work");
let build_dir = data.join("work").join(&key);
world::replace_source(source, &build_dir)?; world::replace_source(source, &build_dir)?;
let manifest = build_dir.join("Cargo.toml"); let manifest = build_dir.join("Cargo.toml");
std::fs::write(&manifest, prepare_manifest(&std::fs::read_to_string(&manifest)?, Some(&data.join("sdk")))?)?; std::fs::write(&manifest, prepare_manifest(&std::fs::read_to_string(&manifest)?, Some(&self.world.public_dir(ID).join("sdk")))?)?;
let mut cmd = Command::new("cargo"); let mut cmd = Command::new("cargo");
cmd.arg("build") cmd.arg("build")
@@ -99,8 +99,12 @@ impl Cargo {
} }
/// The native `cargo` actor. Builds run one at a time, in order. /// The native `cargo` actor. Builds run one at a time, in order.
/// Replies (messages with `reply_to`) get no answer.
pub async fn run_actor(self, broker: Arc<Broker>, mut mailbox: mpsc::Receiver<Envelope>) { pub async fn run_actor(self, broker: Arc<Broker>, mut mailbox: mpsc::Receiver<Envelope>) {
while let Some(env) = mailbox.recv().await { while let Some(env) = mailbox.recv().await {
if env.reply_to.is_some() {
continue;
}
let body = self.handle(&broker, &env).await.unwrap_or_else(|e| { let body = self.handle(&broker, &env).await.unwrap_or_else(|e| {
json!({"type": "build_failed", "source": env.body.get("source"), "errors": format!("{e:#}")}) json!({"type": "build_failed", "source": env.body.get("source"), "errors": format!("{e:#}")})
}); });
@@ -124,7 +128,6 @@ impl Cargo {
"warnings": out.warnings, "warnings": out.warnings,
})) }))
} }
Some("shutdown") => Ok(json!({"type": "ignored"})),
other => bail!("unknown request {other:?}. Known: build"), other => bail!("unknown request {other:?}. Known: build"),
} }
} }
@@ -197,12 +200,3 @@ fn strip_progress(stderr: &str) -> String {
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join("\n") .join("\n")
} }
fn fxhash(bytes: &[u8]) -> u64 {
let mut h: u64 = 0xcbf29ce484222325;
for b in bytes {
h ^= *b as u64;
h = h.wrapping_mul(0x100000001b3);
}
h
}
cli/src/console.rs
+38 -178
View File
@@ -3,205 +3,65 @@
//! - `@<actor> <json>` sends a JSON body. `@<actor> <text>` sends //! - `@<actor> <json>` sends a JSON body. `@<actor> <text>` sends
//! `{"type": "text", "text": ...}`. //! `{"type": "text", "text": ...}`.
//! - A line without `@` goes to `agent` as text. //! - A line without `@` goes to `agent` as text.
//! - `/deploy <actor> <folder>` builds a Rust folder with `cargo`, then spawns //! - `/help`, `/quit`. EOF on stdin also quits.
//! or upgrades the actor.
//! - `/list`, `/sleep <seconds>`, `/help`, `/quit`.
use std::collections::HashMap; use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use anyhow::{Result, anyhow, bail};
use serde_json::{Value, json}; use serde_json::{Value, json};
use tokio::io::{AsyncBufReadExt, BufReader}; use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::sync::{mpsc, oneshot}; use tokio::sync::mpsc;
use crate::broker::{Broker, Envelope}; use crate::broker::{Broker, Envelope};
pub const ID: &str = "console"; pub const ID: &str = "console";
const HELP: &str = "\ const HELP: &str = "\
@<actor> <json> send a JSON body to an actor @<actor> <json> send a JSON body to an actor
@<actor> <text> send {\"type\":\"text\",\"text\":...} @<actor> <text> send {\"type\":\"text\",\"text\":...}
<text> send text to the agent <text> send text to the agent
/deploy <actor> <dir> build a Rust folder with cargo, then spawn or upgrade /quit stop all actors and exit";
/list list actors
/sleep <seconds> wait (for scripts)
/await <actor> <secs> wait for the next message from an actor (for scripts)
/quit stop all actors and exit";
pub struct Console {
broker: Arc<Broker>,
waiters: Mutex<HashMap<String, oneshot::Sender<Envelope>>>,
seq: AtomicU64,
/// Number of messages received per sender, for `/await`.
seen: Mutex<HashMap<String, u64>>,
arrived: tokio::sync::Notify,
}
/// Print a message: text bodies as text, everything else as JSON.
fn show(env: &Envelope) { fn show(env: &Envelope) {
let reply = if env.reply_to.is_some() { " ↩" } else { "" }; let reply = if env.reply_to.is_some() { " ↩" } else { "" };
let b = &env.body; match env.body["text"].as_str() {
if b["type"] == "text" Some(text) if env.body["type"] == "text" => println!("[{}{reply}] {text}", env.from),
&& let Some(t) = b["text"].as_str() _ => println!("[{}{reply}] {}", env.from, env.body),
{
println!("[{}{reply}] {t}", env.from);
return;
}
let mut compact = b.clone();
let mut extra = Vec::new();
if let Some(obj) = compact.as_object_mut() {
for key in ["errors", "warnings", "reason"] {
if let Some(Value::String(s)) = obj.get(key)
&& s.contains('\n')
{
extra.push((key, s.clone()));
obj.remove(key);
}
}
}
println!("[{}{reply}] {compact}", env.from);
for (k, s) in extra {
println!(" {k}:\n{}", s.lines().map(|l| format!(" {l}")).collect::<Vec<_>>().join("\n"));
} }
} }
impl Console { /// Print every message that arrives for the console.
pub fn new(broker: Arc<Broker>) -> Arc<Self> { pub async fn run_mailbox(mut mailbox: mpsc::Receiver<Envelope>) {
Arc::new(Self { while let Some(env) = mailbox.recv().await {
broker, show(&env);
waiters: Mutex::new(HashMap::new()),
seq: AtomicU64::new(1),
seen: Mutex::new(HashMap::new()),
arrived: tokio::sync::Notify::new(),
})
} }
}
pub async fn run_mailbox(self: Arc<Self>, mut mailbox: mpsc::Receiver<Envelope>) { /// Read lines until EOF or `/quit`, and send them as messages.
while let Some(env) = mailbox.recv().await { pub async fn run_input(broker: Arc<Broker>) {
show(&env); println!("Sausage console. Type /help for commands.");
*self.seen.lock().unwrap().entry(env.from.clone()).or_default() += 1; let mut lines = BufReader::new(tokio::io::stdin()).lines();
self.arrived.notify_waiters(); while let Ok(Some(line)) = lines.next_line().await {
if let Some(r) = &env.reply_to let line = line.trim();
&& let Some(tx) = self.waiters.lock().unwrap().remove(r) let (to, body) = match line {
{ "" => continue,
let _ = tx.send(env); "/quit" => break,
} "/help" => {
} println!("{HELP}");
}
fn send(&self, to: &str, body: Value) -> String {
let own = format!("c{}", self.seq.fetch_add(1, Ordering::Relaxed));
self.broker.send(ID, Some(to), Some(&own), None, body)
}
async fn request(&self, to: &str, body: Value, timeout: Duration) -> Result<Envelope> {
let own = format!("c{}", self.seq.fetch_add(1, Ordering::Relaxed));
let (tx, rx) = oneshot::channel();
self.waiters.lock().unwrap().insert(format!("{ID}/{own}"), tx);
self.broker.send(ID, Some(to), Some(&own), None, body);
tokio::time::timeout(timeout, rx)
.await
.map_err(|_| anyhow!("no reply from {to} within {timeout:?}"))?
.map_err(|_| anyhow!("reply channel closed"))
}
async fn deploy(&self, actor: &str, dir: &str) -> Result<()> {
let path = std::path::absolute(dir)?;
let source = path.display().to_string();
println!("· building {source} …");
let built = self.request(crate::cargo_actor::ID, json!({"type": "build", "source": source}), Duration::from_secs(900)).await?;
if built.body["type"] != "built" {
bail!("build failed");
}
let exists = self.broker.meta(actor).is_some_and(|m| m.version.is_some());
let ty = if exists { "upgrade" } else { "spawn" };
println!("· {ty} {actor} …");
let body = json!({
"type": ty,
"actor": actor,
"wasm": built.body["wasm"],
"source": source,
"built_by": built.body["built_by"],
});
let r = self.request("broker", body, Duration::from_secs(300)).await?;
if r.body["type"] == "error" {
bail!("{ty} failed");
}
Ok(())
}
/// Read commands until EOF or `/quit`.
pub async fn run_input(self: Arc<Self>) {
println!("Sausage console. Type /help for commands.");
let mut lines = BufReader::new(tokio::io::stdin()).lines();
while let Ok(Some(line)) = lines.next_line().await {
let line = line.trim();
if line.is_empty() || line.starts_with('#') {
continue; continue;
} }
if let Err(e) = self.command(line).await { _ => match line.strip_prefix('@') {
if e.to_string() == "quit" { Some(rest) => {
break; let (to, text) = rest.split_once(char::is_whitespace).unwrap_or((rest, ""));
} let body = match serde_json::from_str::<Value>(text.trim()) {
println!("! {e:#}"); Ok(v @ Value::Object(_)) => v,
} _ => json!({"type": "text", "text": text.trim()}),
}
}
async fn command(&self, line: &str) -> Result<()> {
if let Some(rest) = line.strip_prefix('@') {
let (to, text) = rest.split_once(char::is_whitespace).unwrap_or((rest, ""));
let text = text.trim();
let body = match serde_json::from_str::<Value>(text) {
Ok(v @ Value::Object(_)) => v,
_ => json!({"type": "text", "text": text}),
};
self.send(to, body);
return Ok(());
}
if let Some(cmd) = line.strip_prefix('/') {
let mut parts = cmd.split_whitespace();
match parts.next() {
Some("help") => println!("{HELP}"),
Some("quit") => bail!("quit"),
Some("list") => {
self.send("broker", json!({"type": "list"}));
}
Some("sleep") => {
let secs: f64 = parts.next().and_then(|s| s.parse().ok()).unwrap_or(1.0);
tokio::time::sleep(Duration::from_secs_f64(secs)).await;
}
Some("await") => {
let from = parts.next().unwrap_or("agent").to_string();
let secs: f64 = parts.next().and_then(|s| s.parse().ok()).unwrap_or(60.0);
let count = |c: &Self| c.seen.lock().unwrap().get(&from).copied().unwrap_or(0);
let before = count(self);
let deadline = tokio::time::Instant::now() + Duration::from_secs_f64(secs);
loop {
let notified = self.arrived.notified();
tokio::pin!(notified);
notified.as_mut().enable();
if count(self) != before {
break;
}
if tokio::time::timeout_at(deadline, notified).await.is_err() {
bail!("no message from {from} within {secs} s");
}
}
}
Some("deploy") => {
let (Some(actor), Some(dir)) = (parts.next(), parts.next()) else {
bail!("usage: /deploy <actor> <dir>");
}; };
self.deploy(actor, dir).await?; (to, body)
} }
_ => bail!("unknown command. Type /help."), None => ("agent", json!({"type": "text", "text": line})),
} },
return Ok(()); };
} broker.send(ID, Some(to), None, None, body);
self.send("agent", json!({"type": "text", "text": line}));
Ok(())
} }
} }
cli/src/fs.rs
+9 -47
View File
@@ -9,9 +9,7 @@
//! //!
//! An open of an existing file without truncate (append, in-place edit) //! An open of an existing file without truncate (append, in-place edit)
//! writes in place, so that appending to a log does not copy the log. //! writes in place, so that appending to a log does not copy the log.
//!
//! A read-only mount of the world's `actor/` folder hides the private
//! `data/` folders of all actors (see `is_private`).
use std::collections::HashMap; use std::collections::HashMap;
use std::path::{Component, Path, PathBuf}; use std::path::{Component, Path, PathBuf};
@@ -42,17 +40,6 @@ struct DirInfo {
root: PathBuf, root: PathBuf,
/// Writable mount: new and truncated files commit on close. /// Writable mount: new and truncated files commit on close.
commit: bool, commit: bool,
/// Read-only mount: private folders are hidden.
hide: bool,
}
/// `<id>/data` and `<id>/versions/<n>/snapshot/data`, relative to `actor/`.
fn is_private(rel: &Path) -> bool {
let c: Vec<_> = rel.components().map(|c| c.as_os_str()).collect();
c.get(1).is_some_and(|n| *n == "data")
|| (c.get(1).is_some_and(|n| *n == "versions")
&& c.get(3).is_some_and(|n| *n == "snapshot")
&& c.get(4).is_some_and(|n| *n == "data"))
} }
struct Pending { struct Pending {
@@ -64,14 +51,12 @@ struct Pending {
pub struct FsExt { pub struct FsExt {
mounts: Vec<Mount>, mounts: Vec<Mount>,
dirs: HashMap<u32, DirInfo>, dirs: HashMap<u32, DirInfo>,
/// Directory streams of hiding mounts, and the directory they list.
streams: HashMap<u32, DirInfo>,
pending: HashMap<u32, Pending>, pending: HashMap<u32, Pending>,
} }
impl FsExt { impl FsExt {
pub fn new(mounts: Vec<Mount>) -> Self { pub fn new(mounts: Vec<Mount>) -> Self {
Self { mounts, dirs: HashMap::new(), streams: HashMap::new(), pending: HashMap::new() } Self { mounts, dirs: HashMap::new(), pending: HashMap::new() }
} }
/// Commit every open write. Use this when the guest exits normally, /// Commit every open write. Use this when the guest exits normally,
@@ -127,17 +112,6 @@ pub struct ActorFs<'a> {
pub ext: &'a mut FsExt, pub ext: &'a mut FsExt,
} }
impl ActorFs<'_> {
/// Whether `path` (relative to the directory `fd`) must look absent:
/// an uncommitted file, or a private folder in a hiding mount.
fn blocked(&self, fd: &Resource<types::Descriptor>, path: &str) -> bool {
is_tmp_name(path)
|| self.ext.dirs.get(&fd.rep()).is_some_and(|d| {
d.hide && resolve(d, path).is_some_and(|p| p.strip_prefix(&d.root).is_ok_and(is_private))
})
}
}
pub struct ActorFsData; pub struct ActorFsData;
impl HasData for ActorFsData { impl HasData for ActorFsData {
@@ -151,7 +125,7 @@ impl preopens::Host for ActorFs<'_> {
if let Some(m) = self.ext.mounts.iter().find(|m| &m.guest == name) { if let Some(m) = self.ext.mounts.iter().find(|m| &m.guest == name) {
self.ext.dirs.insert( self.ext.dirs.insert(
r.rep(), r.rep(),
DirInfo { host: m.host.clone(), root: m.host.clone(), commit: m.writable, hide: !m.writable }, DirInfo { host: m.host.clone(), root: m.host.clone(), commit: m.writable },
); );
} }
} }
@@ -230,12 +204,7 @@ impl HostDescriptor for ActorFs<'_> {
&mut self, &mut self,
fd: Resource<types::Descriptor>, fd: Resource<types::Descriptor>,
) -> FsResult<Resource<types::DirectoryEntryStream>> { ) -> FsResult<Resource<types::DirectoryEntryStream>> {
let dir = self.ext.dirs.get(&fd.rep()).filter(|d| d.hide).cloned(); HostDescriptor::read_directory(&mut self.inner, fd).await
let stream = HostDescriptor::read_directory(&mut self.inner, fd).await?;
if let Some(dir) = dir {
self.ext.streams.insert(stream.rep(), dir);
}
Ok(stream)
} }
async fn sync(&mut self, fd: Resource<types::Descriptor>) -> FsResult<()> { async fn sync(&mut self, fd: Resource<types::Descriptor>) -> FsResult<()> {
@@ -256,7 +225,7 @@ impl HostDescriptor for ActorFs<'_> {
path_flags: types::PathFlags, path_flags: types::PathFlags,
path: String, path: String,
) -> FsResult<types::DescriptorStat> { ) -> FsResult<types::DescriptorStat> {
if self.blocked(&fd, &path) { if is_tmp_name(&path) {
return Err(ErrorCode::NoEntry.into()); return Err(ErrorCode::NoEntry.into());
} }
HostDescriptor::stat_at(&mut self.inner, fd, path_flags, path).await HostDescriptor::stat_at(&mut self.inner, fd, path_flags, path).await
@@ -292,7 +261,7 @@ impl HostDescriptor for ActorFs<'_> {
oflags: types::OpenFlags, oflags: types::OpenFlags,
flags: types::DescriptorFlags, flags: types::DescriptorFlags,
) -> FsResult<Resource<types::Descriptor>> { ) -> FsResult<Resource<types::Descriptor>> {
if self.blocked(&fd, &path) { if is_tmp_name(&path) {
return Err(ErrorCode::NoEntry.into()); return Err(ErrorCode::NoEntry.into());
} }
let parent = self.ext.dirs.get(&fd.rep()).cloned(); let parent = self.ext.dirs.get(&fd.rep()).cloned();
@@ -345,7 +314,7 @@ impl HostDescriptor for ActorFs<'_> {
} }
async fn readlink_at(&mut self, fd: Resource<types::Descriptor>, path: String) -> FsResult<String> { async fn readlink_at(&mut self, fd: Resource<types::Descriptor>, path: String) -> FsResult<String> {
if self.blocked(&fd, &path) { if is_tmp_name(&path) {
return Err(ErrorCode::NoEntry.into()); return Err(ErrorCode::NoEntry.into());
} }
HostDescriptor::readlink_at(&mut self.inner, fd, path).await HostDescriptor::readlink_at(&mut self.inner, fd, path).await
@@ -441,7 +410,7 @@ impl HostDescriptor for ActorFs<'_> {
path_flags: types::PathFlags, path_flags: types::PathFlags,
path: String, path: String,
) -> FsResult<types::MetadataHashValue> { ) -> FsResult<types::MetadataHashValue> {
if self.blocked(&fd, &path) { if is_tmp_name(&path) {
return Err(ErrorCode::NoEntry.into()); return Err(ErrorCode::NoEntry.into());
} }
HostDescriptor::metadata_hash_at(&mut self.inner, fd, path_flags, path).await HostDescriptor::metadata_hash_at(&mut self.inner, fd, path_flags, path).await
@@ -460,21 +429,14 @@ impl HostDirectoryEntryStream for ActorFs<'_> {
Resource::new_borrow(rep), Resource::new_borrow(rep),
) )
.await?; .await?;
let hidden = |name: &str| {
name.starts_with(TMP_PREFIX)
|| self.ext.streams.get(&rep).is_some_and(|d| {
d.host.join(name).strip_prefix(&d.root).is_ok_and(is_private)
})
};
match entry { match entry {
Some(e) if hidden(&e.name) => continue, Some(e) if e.name.starts_with(TMP_PREFIX) => continue,
other => return Ok(other), other => return Ok(other),
} }
} }
} }
fn drop(&mut self, stream: Resource<types::DirectoryEntryStream>) -> wasmtime::Result<()> { fn drop(&mut self, stream: Resource<types::DirectoryEntryStream>) -> wasmtime::Result<()> {
self.ext.streams.remove(&stream.rep());
HostDirectoryEntryStream::drop(&mut self.inner, stream) HostDirectoryEntryStream::drop(&mut self.inner, stream)
} }
} }
cli/src/lib.rs
+1
View File
@@ -9,6 +9,7 @@ pub mod cargo_actor;
pub mod console; pub mod console;
pub mod fs; pub mod fs;
pub mod runtime; pub mod runtime;
pub mod supervisor;
pub mod world; pub mod world;
use std::path::Path; use std::path::Path;
cli/src/main.rs
+10 -28
View File
@@ -1,14 +1,13 @@
use std::io::Write; use std::io::Write;
use std::path::PathBuf; use std::path::PathBuf;
use std::sync::Arc;
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use clap::{Parser, Subcommand}; use clap::{Parser, Subcommand};
use include_dir::{Dir, include_dir}; use include_dir::{Dir, include_dir};
use serde_json::json; use serde_json::json;
use sausage::world::{self, ActorMeta, VersionMeta, World}; use sausage::world::{ActorMeta, World};
use sausage::{Broker, Host, cargo_actor, console, runtime}; use sausage::{Host, cargo_actor, console};
/// The source of the agent actor. `init` builds it like any other actor. /// The source of the agent actor. `init` builds it like any other actor.
static AGENT_SOURCE: Dir<'_> = include_dir!("$CARGO_MANIFEST_DIR/../agent"); static AGENT_SOURCE: Dir<'_> = include_dir!("$CARGO_MANIFEST_DIR/../agent");
@@ -72,45 +71,28 @@ async fn init(dir: PathBuf) -> Result<()> {
let console = ActorMeta { subscriptions: ["agent".to_string()].into(), ..world.read_meta(console::ID) }; let console = ActorMeta { subscriptions: ["agent".to_string()].into(), ..world.read_meta(console::ID) };
world.write_meta(console::ID, &console)?; world.write_meta(console::ID, &console)?;
// The agent's private configuration. Only the agent can read /data. // The agent's configuration, in its /data.
world.ensure_actor_dirs("agent")?; world.ensure_actor_dirs("agent")?;
let config = json!({"api": api, "base_url": base_url.trim_end_matches('/'), "model": model, "token": token}); let config = json!({"api": api, "base_url": base_url.trim_end_matches('/'), "model": model, "token": token});
let config_path = world.data_dir("agent").join("config.json"); let config_path = world.data_dir("agent").join("config.json");
std::fs::write(&config_path, serde_json::to_string_pretty(&config)?)?; std::fs::write(&config_path, serde_json::to_string_pretty(&config)?)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
std::fs::set_permissions(&config_path, std::fs::Permissions::from_mode(0o600))?;
}
let source = world.scratch_dir().join("seed").join("agent"); // The agent is an ordinary actor folder: `run` finds its `actor.wasm`,
// tests it, and installs it as version 1.
let source = world.source_dir("agent");
std::fs::create_dir_all(&source)?; std::fs::create_dir_all(&source)?;
AGENT_SOURCE.extract(&source).context("extract agent source")?; AGENT_SOURCE.extract(&source).context("extract agent source")?;
println!("Building the agent actor (the first build takes a few minutes) …"); println!("Building the agent actor (the first build takes a few minutes) …");
let out = cargo_actor::Cargo::new(world.clone())?.build(&source).await.context("build agent")?; let out = cargo_actor::Cargo::new(world.clone())?.build(&source).await.context("build agent")?;
println!("Built in {:.1} s.", out.seconds); std::fs::copy(&out.wasm, world.wasm_path("agent"))?;
println!("Built in {:.1} s. Start the world with: sausage run {}", out.seconds, dir.display());
let broker = Broker::new(world.clone(), runtime::Runtime::new()?)?;
let meta = VersionMeta {
version: 1,
wasm: world.global_path(&out.wasm).unwrap_or_default(),
source: Some(source.display().to_string()),
built_by: Some(cargo_actor::ID.into()),
requested_by: "init".into(),
created_at: world::now(),
rejected: None,
};
broker.create_actor("agent", &out.wasm, Some(&source), meta, false).await?;
println!("World ready. Start it with: sausage run {}", dir.display());
Ok(()) Ok(())
} }
async fn run(dir: PathBuf) -> Result<()> { async fn run(dir: PathBuf) -> Result<()> {
let host = Host::start(&dir).await?; let host = Host::start(&dir).await?;
let rx = host.mailbox(console::ID); tokio::spawn(console::run_mailbox(host.mailbox(console::ID)));
let console = console::Console::new(host.broker.clone()); console::run_input(host.broker.clone()).await;
tokio::spawn(Arc::clone(&console).run_mailbox(rx));
console.run_input().await;
println!("Stopping actors …"); println!("Stopping actors …");
host.stop().await; host.stop().await;
Ok(()) Ok(())
cli/src/supervisor.rs
+296
View File
@@ -0,0 +1,296 @@
//! The supervisor of one WASM actor: it runs the instance, feeds it the
//! mailbox, routes its output, restarts it after a crash, and swaps versions.
//!
//! The broker talks to a supervisor only through `Control` requests.
use std::sync::Arc;
use std::time::{Duration, Instant};
use anyhow::{Result, bail};
use serde_json::{Value, json};
use tokio::sync::{mpsc, oneshot};
use wasmtime::component::Component;
use crate::broker::{Broker, Envelope, SHUTDOWN_DEADLINE, shutdown_line};
use crate::runtime::{Exit, Instance, InstanceConfig};
/// A new version that crashes this often within `CRASH_WINDOW` after an
/// upgrade is rolled back automatically.
const CRASH_LIMIT: usize = 3;
const CRASH_WINDOW: Duration = Duration::from_secs(30);
pub type Reply = oneshot::Sender<Result<Value>>;
pub enum Control {
/// Swap to `version` (already in `versions/`, compiled as `component`).
Upgrade { version: u64, component: Component, requester: String, reply: Reply },
Rollback { version: u64, reply: Reply },
Stop { reply: Reply },
Start { reply: Reply },
Shutdown { reply: Reply },
}
/// The time after an upgrade in which crashes count toward a rollback.
struct Probation {
previous: u64,
since: Instant,
crashes: usize,
requester: String,
}
pub struct Supervisor {
broker: Arc<Broker>,
id: String,
version: u64,
running: bool,
mailbox: mpsc::Receiver<Envelope>,
control: mpsc::Receiver<Control>,
component: Option<Component>,
instance: Option<Instance>,
started_at: Instant,
restarts: u32,
restart_at: Option<Instant>,
probation: Option<Probation>,
/// A control request that arrived during a blocked delivery.
next_control: Option<Control>,
}
async fn next_line(instance: &mut Option<Instance>) -> Option<String> {
match instance {
Some(i) => i.stdout.recv().await,
None => std::future::pending().await,
}
}
async fn sleep_until(t: Option<Instant>) {
match t {
Some(t) => tokio::time::sleep_until(t.into()).await,
None => std::future::pending().await,
}
}
impl Supervisor {
pub fn new(
broker: Arc<Broker>,
id: String,
version: u64,
running: bool,
mailbox: mpsc::Receiver<Envelope>,
control: mpsc::Receiver<Control>,
) -> Self {
Self {
broker,
id,
version,
running,
mailbox,
control,
component: None,
instance: None,
started_at: Instant::now(),
restarts: 0,
restart_at: None,
probation: None,
next_control: None,
}
}
pub async fn run(mut self) {
loop {
if let Some(ctl) = self.next_control.take() {
if self.handle(ctl).await {
break;
}
continue;
}
if self.running && self.instance.is_none() && self.restart_at.is_none_or(|t| Instant::now() >= t) {
self.restart_at = None;
let _ = self.start().await;
}
let has_instance = self.instance.is_some();
tokio::select! {
ctl = self.control.recv() => {
let Some(ctl) = ctl else { break };
if self.handle(ctl).await {
break;
}
}
msg = self.mailbox.recv(), if has_instance => {
let Some(msg) = msg else { break };
self.deliver(msg).await;
}
line = next_line(&mut self.instance) => match line {
Some(line) => self.broker.route_line(&self.id, &line),
None => {
let exit = self.instance.as_mut().unwrap().wait().await;
self.instance = None;
self.on_exit(exit).await;
}
},
_ = sleep_until(self.restart_at), if !has_instance && self.running => {}
}
}
}
/// Write a message to the instance's stdin, and route its output
/// meanwhile. If a control request arrives before the actor reads the
/// message, the instance is killed (a partly written line would corrupt
/// the next message) and the request runs.
async fn deliver(&mut self, msg: Envelope) {
let Some(inst) = self.instance.as_mut() else { return };
let (broker, id, line) = (self.broker.clone(), self.id.clone(), msg.to_line());
let stuck = tokio::select! {
written = inst.send_line(&line, |l| broker.route_line(&id, &l)) => match written {
Ok(()) => return,
// The instance is ending. The main loop sees its closed stdout.
Err(_) => false,
},
Some(ctl) = self.control.recv() => {
self.next_control = Some(ctl);
true
}
};
if msg.from != "broker" {
let body = json!({"type": "undeliverable", "reason": "the actor stopped", "to": self.id});
self.broker.send("broker", Some(&msg.from), None, Some(&msg.id), body);
}
if stuck && let Some(inst) = &self.instance {
self.broker.log(&self.id, "killed: it did not read stdin when the broker needed it");
inst.kill();
}
}
/// Start an instance of the current version. If that fails, the actor
/// stays stopped until the next start, upgrade, or rollback.
async fn start(&mut self) -> Result<()> {
let started = self.try_start();
match &started {
Ok(()) => self.started_at = Instant::now(),
Err(e) => {
self.broker.log(&self.id, &format!("cannot start version {}: {e:#}", self.version));
self.running = false;
}
}
started
}
fn try_start(&mut self) -> Result<()> {
if self.component.is_none() {
self.component = Some(self.broker.load_version(&self.id, self.version)?);
}
let world = &self.broker.world;
let cfg = InstanceConfig {
mounts: self.broker.mounts(&self.id),
allow_http: true,
stderr_log: world.actor_dir(&self.id).join("stderr.log"),
env: vec![
("SAUSAGE_ACTOR".into(), self.id.clone()),
("SAUSAGE_VERSION".into(), self.version.to_string()),
],
};
self.instance = Some(self.broker.runtime.start(self.component.as_ref().unwrap(), cfg)?);
Ok(())
}
/// Stop the instance gracefully. Its last messages still get routed.
async fn stop(&mut self) {
let Some(inst) = self.instance.take() else { return };
let (broker, id) = (self.broker.clone(), self.id.clone());
let exit = inst.stop(&shutdown_line(), SHUTDOWN_DEADLINE, |line| broker.route_line(&id, &line)).await;
self.broker.log(&self.id, &format!("stopped: {exit}"));
}
async fn on_exit(&mut self, exit: Exit) {
self.broker.log(&self.id, &format!("instance ended unexpectedly: {exit}"));
if let Some(p) = &mut self.probation
&& p.since.elapsed() < CRASH_WINDOW
{
p.crashes += 1;
if p.crashes >= CRASH_LIMIT {
let (from, to, requester) = (self.version, p.previous, p.requester.clone());
self.broker.log(&self.id, &format!("version {from} crashed {CRASH_LIMIT} times after the upgrade; rolling back to {to}"));
let body = match self.rollback(to).await {
Ok(_) => json!({"type": "rolled_back", "actor": self.id, "from": from, "to": to,
"reason": "crash loop after upgrade", "last_exit": exit.to_string()}),
Err(e) => json!({"type": "error", "actor": self.id, "reason": format!("automatic rollback failed: {e:#}")}),
};
self.broker.send("broker", Some(&requester), None, None, body);
return;
}
}
// Restart after 0.5 s, doubling up to 30 s; a minute of uptime resets it.
if self.started_at.elapsed() > Duration::from_secs(60) {
self.restarts = 0;
}
let delay = Duration::from_millis(500 << self.restarts.min(6)).min(Duration::from_secs(30));
self.restarts += 1;
self.restart_at = Some(Instant::now() + delay);
}
/// Make `version` current. It is compiled first (unless `component` is
/// given): if that fails, nothing changes. `data/` and `public/` stay as
/// they are.
async fn install(&mut self, version: u64, component: Option<Component>) -> Result<()> {
let component = match component {
Some(c) => c,
None => self.broker.load_version(&self.id, version)?,
};
self.stop().await;
self.broker.install(&self.id, version)?;
self.version = version;
self.component = Some(component);
self.running = true;
self.restarts = 0;
self.restart_at = None;
Ok(())
}
async fn upgrade(&mut self, version: u64, component: Component, requester: String) -> Result<Value> {
let previous = self.version;
self.install(version, Some(component)).await?;
self.probation = Some(Probation { previous, since: Instant::now(), crashes: 0, requester });
if let Err(e) = self.start().await {
self.rollback(previous).await?;
bail!("version {version} did not start ({e:#}); rolled back to {previous}");
}
Ok(json!({"type": "upgraded", "actor": self.id, "version": version, "previous": previous}))
}
async fn rollback(&mut self, version: u64) -> Result<Value> {
let from = self.version;
self.install(version, None).await?;
self.probation = None;
self.start().await?;
Ok(json!({"type": "rolled_back", "actor": self.id, "from": from, "to": version}))
}
/// Handle a control request. Returns `true` when the supervisor ends.
async fn handle(&mut self, ctl: Control) -> bool {
match ctl {
Control::Upgrade { version, component, requester, reply } => {
let _ = reply.send(self.upgrade(version, component, requester).await);
}
Control::Rollback { version, reply } => {
let _ = reply.send(self.rollback(version).await);
}
Control::Stop { reply } => {
self.running = false;
self.stop().await;
let r = self.broker.set_running(&self.id, false);
let _ = reply.send(r.map(|_| json!({"type": "stopped", "actor": self.id})));
}
Control::Start { reply } => {
self.running = true;
self.restart_at = None;
let r = self.broker.set_running(&self.id, true);
let _ = reply.send(r.map(|_| json!({"type": "started", "actor": self.id})));
}
Control::Shutdown { reply } => {
self.stop().await;
let _ = reply.send(Ok(json!({"type": "shut_down"})));
return true;
}
}
false
}
}
cli/src/world.rs
+15 -9
View File
@@ -8,11 +8,9 @@
//! actor.json version, running, subscriptions (broker) //! actor.json version, running, subscriptions (broker)
//! source/ source of the current version //! source/ source of the current version
//! public/ written by the actor, read-only for others //! public/ written by the actor, read-only for others
//! data/ private state; hidden from other actors //! data/ state, writable only by the actor
//! status.json runtime state (broker)
//! stderr.log stderr of the actor and notes of the broker //! stderr.log stderr of the actor and notes of the broker
//! versions/<n>/ actor.wasm, source/, meta.json, snapshot/{data,public} //! versions/<n>/ every version that was installed: actor.wasm, source/, meta.json
//! archive/ removed actors
//! ``` //! ```
use std::collections::BTreeSet; use std::collections::BTreeSet;
@@ -62,9 +60,6 @@ pub struct VersionMeta {
/// The actor that requested the version ("scan" for a file found on disk). /// The actor that requested the version ("scan" for a file found on disk).
pub requested_by: String, pub requested_by: String,
pub created_at: u64, pub created_at: u64,
/// Why the broker did not install this version, if it did not.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub rejected: Option<String>,
} }
pub fn valid_id(id: &str) -> bool { pub fn valid_id(id: &str) -> bool {
@@ -126,8 +121,6 @@ impl World {
} }
for id in NATIVE { for id in NATIVE {
world.ensure_actor_dirs(id)?; world.ensure_actor_dirs(id)?;
let status = serde_json::json!({"id": id, "kind": "native", "state": "built-in"});
write_atomic(&world.actor_dir(id).join("status.json"), serde_json::to_string_pretty(&status)?.as_bytes())?;
} }
Ok(world) Ok(world)
} }
@@ -161,6 +154,19 @@ impl World {
write_atomic(&self.actor_dir(id).join("actor.json"), serde_json::to_string_pretty(meta)?.as_bytes()) write_atomic(&self.actor_dir(id).join("actor.json"), serde_json::to_string_pretty(meta)?.as_bytes())
} }
/// Make `actor.wasm` and `source/` a copy of `versions/<version>/`,
/// unless `actor.wasm` already is that version (it came from there).
pub fn install_files(&self, id: &str, version: u64) -> Result<()> {
let dir = self.version_dir(id, version);
self.ensure_actor_dirs(id)?;
let wasm = std::fs::read(dir.join("actor.wasm"))?;
if std::fs::read(self.wasm_path(id)).ok().as_ref() == Some(&wasm) {
return Ok(());
}
write_atomic(&self.wasm_path(id), &wasm)?;
replace_tree(&dir.join("source"), &self.source_dir(id))
}
/// Version numbers in `versions/`, ascending. /// Version numbers in `versions/`, ascending.
pub fn versions(&self, id: &str) -> Vec<u64> { pub fn versions(&self, id: &str) -> Vec<u64> {
let mut v: Vec<u64> = std::fs::read_dir(self.versions_dir(id)) let mut v: Vec<u64> = std::fs::read_dir(self.versions_dir(id))
cli/tests/actors/toolbox/src/main.rs
-2
View File
@@ -146,8 +146,6 @@ fn main() {
} }
Err(error) => io_error(error), Err(error) => io_error(error),
}, },
Some("crash") if version == "1" => panic!("asked to crash"),
Some("crash") => json!({"type": "replayed", "version": version}),
_ => json!({"type": "error", "error": "unknown message"}), _ => json!({"type": "error", "error": "unknown message"}),
}; };
reply(&mut out, &message, response); reply(&mut out, &message, response);
cli/tests/agent.rs
+23 -115
View File
@@ -9,7 +9,7 @@ use std::process::Command;
use std::sync::OnceLock; use std::sync::OnceLock;
use std::time::Duration; use std::time::Duration;
use common::{create_world, receive_replies, receive_reply, request, source_with_test, start}; use common::{create_world, receive_message, receive_reply, request, source_with_test, start};
use sausage::{Envelope, Host, World}; use sausage::{Envelope, Host, World};
use serde_json::{Value, json}; use serde_json::{Value, json};
use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::io::{AsyncReadExt, AsyncWriteExt};
@@ -319,6 +319,13 @@ async fn start_agent(
spawned.body["type"], "spawned", spawned.body["type"], "spawned",
"agent spawn failed: {spawned:?}" "agent spawn failed: {spawned:?}"
); );
let subscribed = request(
&host,
&mut mailbox,
json!({"type": "subscribe", "actor": "agent"}),
)
.await;
assert_eq!(subscribed.body["type"], "subscribed");
(world, host, mailbox) (world, host, mailbox)
} }
@@ -332,11 +339,12 @@ fn send_text(host: &Host, text: &str) -> String {
) )
} }
async fn assert_answer(mailbox: &mut mpsc::Receiver<Envelope>, id: &str, text: &str) { async fn assert_published_text(mailbox: &mut mpsc::Receiver<Envelope>, text: &str) {
let reply = receive_reply(mailbox, id, Duration::from_secs(20)).await; let message = receive_message(mailbox, Duration::from_secs(20)).await;
assert_eq!(reply.to, "console"); assert_eq!(message.from, "agent");
assert_eq!(reply.reply_to.as_deref(), Some(id)); assert_eq!(message.to, "console");
assert_eq!(reply.body, json!({"type": "text", "text": text})); assert_eq!(message.reply_to, None);
assert_eq!(message.body, json!({"type": "text", "text": text}));
} }
fn history(world: &World) -> Vec<Value> { fn history(world: &World) -> Vec<Value> {
@@ -439,9 +447,9 @@ async fn text_answers_use_both_inference_apis() {
.await; .await;
let (_, host, mut mailbox) = let (_, host, mut mailbox) =
start_agent(api, &fake, &format!("agent-text-{}", api.name())).await; start_agent(api, &fake, &format!("agent-text-{}", api.name())).await;
let id = send_text(&host, "say hello"); send_text(&host, "say hello");
api.check_request(&fake.next_request().await); api.check_request(&fake.next_request().await);
assert_answer(&mut mailbox, &id, "hello from the fake model").await; assert_published_text(&mut mailbox, "hello from the fake model").await;
host.stop().await; host.stop().await;
} }
} }
@@ -455,7 +463,7 @@ async fn tool_calls_pair_with_results_in_both_inference_apis() {
.await; .await;
let (world, host, mut mailbox) = let (world, host, mut mailbox) =
start_agent(api, &fake, &format!("agent-tools-{}", api.name())).await; start_agent(api, &fake, &format!("agent-tools-{}", api.name())).await;
let id = send_text(&host, "list actors"); send_text(&host, "list actors");
api.check_request(&fake.next_request().await); api.check_request(&fake.next_request().await);
fake.answer(api.text_response("the actor list is ready")) fake.answer(api.text_response("the actor list is ready"))
.await; .await;
@@ -470,8 +478,9 @@ async fn tool_calls_pair_with_results_in_both_inference_apis() {
.iter() .iter()
.any(|entry| entry["name"] == "agent") .any(|entry| entry["name"] == "agent")
); );
assert_answer(&mut mailbox, &id, "the actor list is ready").await; assert_published_text(&mut mailbox, "the actor list is ready").await;
host.stop().await;
let history = history(&world); let history = history(&world);
assert_eq!( assert_eq!(
history history
@@ -482,7 +491,6 @@ async fn tool_calls_pair_with_results_in_both_inference_apis() {
); );
assert_eq!(history[1]["id"], "list-1"); assert_eq!(history[1]["id"], "list-1");
assert_eq!(history[2]["id"], "list-1"); assert_eq!(history[2]["id"], "list-1");
host.stop().await;
} }
} }
@@ -513,8 +521,8 @@ async fn restart_resumes_a_turn_with_completed_tool_history() {
serde_json::from_slice::<Value>( serde_json::from_slice::<Value>(
&std::fs::read(world.data_dir("agent").join("turn.json")).unwrap() &std::fs::read(world.data_dir("agent").join("turn.json")).unwrap()
) )
.unwrap()["active"]["id"], .unwrap(),
id json!(true)
); );
let _ = release.send(()); let _ = release.send(());
fake.answer(api.text_response("resumed answer")).await; fake.answer(api.text_response("resumed answer")).await;
@@ -528,22 +536,9 @@ async fn restart_resumes_a_turn_with_completed_tool_history() {
let resumed = fake.next_request().await; let resumed = fake.next_request().await;
api.check_request(&resumed); api.check_request(&resumed);
assert_tool_pair(&resumed, api, "list-1", "list_dir", &args); assert_tool_pair(&resumed, api, "list-1", "list_dir", &args);
let replies = receive_replies( let started = receive_reply(&mut mailbox, &start_id, Duration::from_secs(20)).await;
&mut mailbox,
&[start_id.clone(), id.clone()],
Duration::from_secs(20),
)
.await;
let started = replies
.iter()
.find(|reply| reply.reply_to.as_deref() == Some(start_id.as_str()))
.unwrap();
assert_eq!(started.body["type"], "started"); assert_eq!(started.body["type"], "started");
let answer = replies assert_published_text(&mut mailbox, "resumed answer").await;
.iter()
.find(|reply| reply.reply_to.as_deref() == Some(id.as_str()))
.unwrap();
assert_eq!(answer.body["text"], "resumed answer");
let history = history(&world); let history = history(&world);
assert_history_tools_have_results(&history); assert_history_tools_have_results(&history);
@@ -558,90 +553,3 @@ async fn restart_resumes_a_turn_with_completed_tool_history() {
host.stop().await; host.stop().await;
} }
} }
#[tokio::test]
async fn queued_text_resumes_after_the_active_turn_across_restart() {
let wait_args = json!({"seconds": 10});
for api in [Api::Anthropic] {
let mut fake = FakeServer::start().await;
fake.answer(api.tool_response("wait-1", "wait_for_messages", &wait_args))
.await;
let (world, host, mut mailbox) =
start_agent(api, &fake, &format!("agent-queue-{}", api.name())).await;
let first_id = send_text(&host, "first turn");
api.check_request(&fake.next_request().await);
let release = fake.hold(api.text_response("discarded response")).await;
let second_id = send_text(&host, "second turn");
let second_request = fake.next_request().await;
api.check_request(&second_request);
let output = assert_tool_pair(
&second_request,
api,
"wait-1",
"wait_for_messages",
&wait_args,
);
let queued: Value = serde_json::from_str(&output).unwrap();
assert!(
queued["messages"]
.as_array()
.unwrap()
.iter()
.any(|message| message["message"]["id"] == second_id)
);
let turn: Value = serde_json::from_slice(
&std::fs::read(world.data_dir("agent").join("turn.json")).unwrap(),
)
.unwrap();
assert_eq!(turn["queue"][0]["id"], second_id);
let stopped = request(
&host,
&mut mailbox,
json!({"type": "stop", "actor": "agent"}),
)
.await;
assert_eq!(stopped.body["type"], "stopped");
let _ = release.send(());
fake.answer(api.text_response("first answer")).await;
fake.answer(api.text_response("second answer")).await;
let start_id = host.broker.send(
"console",
Some("broker"),
None,
None,
json!({"type": "start", "actor": "agent"}),
);
let resumed_first = fake.next_request().await;
api.check_request(&resumed_first);
assert!(resumed_first.body.to_string().contains("first turn"));
let next_turn = fake.next_request().await;
api.check_request(&next_turn);
assert!(next_turn.body.to_string().contains("second turn"));
let replies = receive_replies(
&mut mailbox,
&[start_id.clone(), first_id.clone(), second_id.clone()],
Duration::from_secs(20),
)
.await;
let started = replies
.iter()
.find(|reply| reply.reply_to.as_deref() == Some(start_id.as_str()))
.unwrap();
assert_eq!(started.body["type"], "started");
let first_position = replies
.iter()
.position(|reply| reply.reply_to.as_deref() == Some(first_id.as_str()))
.unwrap();
let second_position = replies
.iter()
.position(|reply| reply.reply_to.as_deref() == Some(second_id.as_str()))
.unwrap();
assert!(first_position < second_position);
assert_eq!(replies[first_position].body["text"], "first answer");
assert_eq!(replies[second_position].body["text"], "second answer");
assert_history_tools_have_results(&history(&world));
host.stop().await;
}
}
cli/tests/common/mod.rs
-23
View File
@@ -246,26 +246,3 @@ pub async fn upgrade_actor(
) )
.await .await
} }
pub async fn wait_for_parked(world: &World, actor: &str, message_id: &str) {
tokio::time::timeout(Duration::from_secs(5), async {
loop {
let status_path = world.actor_dir(actor).join("status.json");
let parked_id = std::fs::read_to_string(status_path)
.ok()
.and_then(|text| serde_json::from_str::<Value>(&text).ok())
.and_then(|status| {
status
.pointer("/parked_message/id")
.and_then(Value::as_str)
.map(str::to_string)
});
if parked_id.as_deref() == Some(message_id) {
return;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("actor did not park the crashed message");
}
cli/tests/host.rs
+75 -210
View File
@@ -1,6 +1,6 @@
mod common; mod common;
use std::time::{Duration, SystemTime, UNIX_EPOCH}; use std::time::Duration;
use common::*; use common::*;
use sausage::fs::TMP_PREFIX; use sausage::fs::TMP_PREFIX;
@@ -48,13 +48,6 @@ async fn receive_shutdown_messages(
.expect("shutdown output or lifecycle reply did not arrive"); .expect("shutdown output or lifecycle reply did not arrive");
} }
fn epoch_second() -> u64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs()
}
#[tokio::test] #[tokio::test]
async fn routes_replies_publishes_and_reports_undeliverable() { async fn routes_replies_publishes_and_reports_undeliverable() {
let world = create_world("routing"); let world = create_world("routing");
@@ -112,6 +105,29 @@ async fn routes_replies_publishes_and_reports_undeliverable() {
host.stop().await; host.stop().await;
} }
#[tokio::test]
async fn broker_does_not_answer_messages_with_reply_to() {
let world = create_world("broker-reply-to");
let (host, mut mailbox) = start(&world).await;
let scanned = request(&host, &mut mailbox, json!({"type": "scan"})).await;
assert_eq!(scanned.body["type"], "scanned");
host.broker.send(
"console",
Some("broker"),
None,
Some("upstream-message"),
json!({"type": "scan"}),
);
assert!(
tokio::time::timeout(Duration::from_secs(1), mailbox.recv())
.await
.is_err(),
"broker replied to a message that already had reply_to"
);
host.stop().await;
}
#[tokio::test] #[tokio::test]
async fn filesystem_permissions_and_public_sharing() { async fn filesystem_permissions_and_public_sharing() {
let (_, host, mut mailbox) = filesystem_world("filesystem-permissions").await; let (_, host, mut mailbox) = filesystem_world("filesystem-permissions").await;
@@ -148,6 +164,47 @@ async fn filesystem_permissions_and_public_sharing() {
host.stop().await; host.stop().await;
} }
#[tokio::test]
async fn actors_can_read_but_not_write_other_actor_data() {
let (_, host, mut mailbox) = filesystem_world("actor-data-sharing").await;
ask_type(
&host,
&mut mailbox,
"writer",
json!({"type": "write", "path": "/data/x", "text": "shared"}),
"ok",
)
.await;
let shared = ask_type(
&host,
&mut mailbox,
"reader",
json!({"type": "read", "path": "/actor/writer/data/x"}),
"content",
)
.await;
assert_eq!(shared.body["text"], "shared");
ask_type(
&host,
&mut mailbox,
"reader",
json!({"type": "write", "path": "/actor/writer/data/x", "text": "changed"}),
"error",
)
.await;
let unchanged = ask_type(
&host,
&mut mailbox,
"writer",
json!({"type": "read", "path": "/data/x"}),
"content",
)
.await;
assert_eq!(unchanged.body["text"], "shared");
host.stop().await;
}
#[tokio::test] #[tokio::test]
async fn filesystem_hides_new_and_truncated_writes_until_close() { async fn filesystem_hides_new_and_truncated_writes_until_close() {
let (_, host, mut mailbox) = filesystem_world("filesystem-commit").await; let (_, host, mut mailbox) = filesystem_world("filesystem-commit").await;
@@ -274,42 +331,6 @@ async fn filesystem_append_is_visible_before_close() {
host.stop().await; host.stop().await;
} }
#[tokio::test]
async fn filesystem_crash_discards_an_open_truncated_write() {
let (world, host, mut mailbox) = filesystem_world("filesystem-crash").await;
ask(
&host,
&mut mailbox,
"writer",
json!({"type": "write", "path": "/public/crash.txt", "text": "old"}),
)
.await;
ask(
&host,
&mut mailbox,
"writer",
json!({"type": "open", "path": "/public/crash.txt", "text": "uncommitted"}),
)
.await;
let crash_id = host.broker.send(
"console",
Some("writer"),
None,
None,
json!({"type": "crash"}),
);
wait_for_parked(&world, "writer", &crash_id).await;
let after_crash = ask(
&host,
&mut mailbox,
"reader",
json!({"type": "read", "path": "/actor/writer/public/crash.txt"}),
)
.await;
assert_eq!(after_crash.body["text"], "old");
host.stop().await;
}
#[tokio::test] #[tokio::test]
async fn filesystem_rejects_paths_that_escape_data() { async fn filesystem_rejects_paths_that_escape_data() {
let (_, host, mut mailbox) = filesystem_world("filesystem-path-escape").await; let (_, host, mut mailbox) = filesystem_world("filesystem-path-escape").await;
@@ -351,6 +372,8 @@ async fn upgrades_test_candidates_keep_old_versions_and_queue_messages_in_order(
assert_eq!(failed.body["type"], "error"); assert_eq!(failed.body["type"], "error");
let old_version = ask(&host, &mut mailbox, "queue", json!({"type": "version"})).await; let old_version = ask(&host, &mut mailbox, "queue", json!({"type": "version"})).await;
assert_eq!(old_version.body["version"], "1"); assert_eq!(old_version.body["version"], "1");
// A version that fails its tests never enters versions/.
assert!(!world.actor_dir("queue").join("versions/2").exists());
let upgrade_id = host.broker.send( let upgrade_id = host.broker.send(
"console", "console",
@@ -408,72 +431,14 @@ async fn upgrades_test_candidates_keep_old_versions_and_queue_messages_in_order(
assert_eq!(envelope.from, "queue"); assert_eq!(envelope.from, "queue");
assert_eq!(envelope.body["type"], "ordered"); assert_eq!(envelope.body["type"], "ordered");
assert_eq!(envelope.body["n"], json!(index + 1)); assert_eq!(envelope.body["n"], json!(index + 1));
assert_eq!(envelope.body["version"], "3"); assert_eq!(envelope.body["version"], "2");
} }
host.stop().await; host.stop().await;
} }
#[tokio::test] #[tokio::test]
async fn redelivers_a_crashed_message_after_the_next_upgrade() { async fn rolls_back_a_crash_loop_to_the_previous_version_without_restoring_data() {
let world = create_world("redelivery");
let wasm = toolbox_wasm().clone();
let (host, mut mailbox) = start(&world).await;
let source = passing_source(&world, "source");
assert_eq!(
spawn_actor(&host, &mut mailbox, "parked", &wasm, &source)
.await
.body["type"],
"spawned"
);
let crash_id = host.broker.send(
"console",
Some("parked"),
None,
None,
json!({"type": "crash"}),
);
wait_for_parked(&world, "parked", &crash_id).await;
assert!(matches!(
mailbox.try_recv(),
Err(tokio::sync::mpsc::error::TryRecvError::Empty)
));
let upgrade_id = host.broker.send(
"console",
Some("broker"),
None,
None,
json!({"type": "upgrade", "actor": "parked", "wasm": wasm.display().to_string(), "source": source.display().to_string()}),
);
let mut upgraded = None;
let mut replayed = None;
tokio::time::timeout(Duration::from_secs(15), async {
while upgraded.is_none() || replayed.is_none() {
let envelope = mailbox.recv().await.expect("console mailbox closed");
if envelope.reply_to.as_deref() == Some(upgrade_id.as_str()) {
upgraded = Some(envelope);
} else if envelope.reply_to.as_deref() == Some(crash_id.as_str()) {
replayed = Some(envelope);
} else {
panic!("unexpected message during parked delivery: {envelope:?}");
}
}
})
.await
.expect("upgrade did not redeliver the parked message");
assert_eq!(upgraded.unwrap().body["type"], "upgraded");
assert_eq!(
replayed.unwrap().body,
json!({"type": "replayed", "version": "2"})
);
host.stop().await;
}
#[tokio::test]
async fn rolls_back_a_crash_loop_and_restores_its_data_snapshot() {
let world = create_world("crash-loop"); let world = create_world("crash-loop");
let wasm = toolbox_wasm().clone(); let wasm = toolbox_wasm().clone();
let (host, mut mailbox) = start(&world).await; let (host, mut mailbox) = start(&world).await;
@@ -505,7 +470,7 @@ async fn rolls_back_a_crash_loop_and_restores_its_data_snapshot() {
); );
let mut upgraded: Option<Value> = None; let mut upgraded: Option<Value> = None;
let mut rolled_back: Option<Value> = None; let mut rolled_back: Option<Value> = None;
tokio::time::timeout(Duration::from_secs(20), async { tokio::time::timeout(Duration::from_secs(35), async {
while upgraded.is_none() || rolled_back.is_none() { while upgraded.is_none() || rolled_back.is_none() {
let envelope = mailbox.recv().await.expect("console mailbox closed"); let envelope = mailbox.recv().await.expect("console mailbox closed");
let kind = envelope.body["type"].as_str().map(str::to_string); let kind = envelope.body["type"].as_str().map(str::to_string);
@@ -530,20 +495,20 @@ async fn rolls_back_a_crash_loop_and_restores_its_data_snapshot() {
let version = ask(&host, &mut mailbox, "crashloop", json!({"type": "version"})).await; let version = ask(&host, &mut mailbox, "crashloop", json!({"type": "version"})).await;
assert_eq!(version.body["version"], "1"); assert_eq!(version.body["version"], "1");
let restored = ask( let current = ask(
&host, &host,
&mut mailbox, &mut mailbox,
"crashloop", "crashloop",
json!({"type": "read", "path": "/data/state"}), json!({"type": "read", "path": "/data/state"}),
) )
.await; .await;
assert_eq!(restored.body["text"], "stable"); assert_eq!(current.body["text"], "bad-v2");
host.stop().await; host.stop().await;
} }
#[tokio::test] #[tokio::test]
async fn explicit_rollback_restores_the_previous_version_and_snapshot() { async fn explicit_rollback_keeps_current_data() {
let world = create_world("explicit-rollback"); let world = create_world("explicit-rollback");
let wasm = toolbox_wasm().clone(); let wasm = toolbox_wasm().clone();
let (host, mut mailbox) = start(&world).await; let (host, mut mailbox) = start(&world).await;
@@ -581,17 +546,16 @@ async fn explicit_rollback_restores_the_previous_version_and_snapshot() {
assert_eq!(rolled_back.body["type"], "rolled_back"); assert_eq!(rolled_back.body["type"], "rolled_back");
assert_eq!(rolled_back.body["from"], 2); assert_eq!(rolled_back.body["from"], 2);
assert_eq!(rolled_back.body["to"], 1); assert_eq!(rolled_back.body["to"], 1);
assert_eq!(rolled_back.body["restored_state"], true);
let version = ask(&host, &mut mailbox, "recover", json!({"type": "version"})).await; let version = ask(&host, &mut mailbox, "recover", json!({"type": "version"})).await;
assert_eq!(version.body["version"], "1"); assert_eq!(version.body["version"], "1");
let restored = ask( let current = ask(
&host, &host,
&mut mailbox, &mut mailbox,
"recover", "recover",
json!({"type": "read", "path": "/data/state"}), json!({"type": "read", "path": "/data/state"}),
) )
.await; .await;
assert_eq!(restored.body["text"], "version-one"); assert_eq!(current.body["text"], "version-two");
host.stop().await; host.stop().await;
} }
@@ -721,37 +685,6 @@ async fn concurrent_spawns_of_one_id_leave_one_live_instance() {
host.stop().await; host.stop().await;
} }
#[tokio::test]
async fn rollback_preserves_target_and_git_files_in_data_and_public() {
let world = create_world("snapshot-special-directories");
let wasm = toolbox_wasm().clone();
let (host, mut mailbox) = start(&world).await;
let source = passing_source(&world, "source");
spawn_type(&host, &mut mailbox, "snapshot", &wasm, &source).await;
let files = [
("/data/target", "/data/target/x", "data-target"),
("/data/.git", "/data/.git/x", "data-git"),
("/public/target", "/public/target/x", "public-target"),
("/public/.git", "/public/.git/x", "public-git"),
];
for (dir, path, text) in files {
let mkdir = json!({"type": "mkdir", "path": dir});
ask_type(&host, &mut mailbox, "snapshot", mkdir, "ok").await;
let write = json!({"type": "write", "path": path, "text": text});
ask_type(&host, &mut mailbox, "snapshot", write, "ok").await;
}
let upgrade = json!({"type": "upgrade", "actor": "snapshot", "wasm": wasm.display().to_string(), "source": source.display().to_string()});
ask_type(&host, &mut mailbox, "broker", upgrade, "upgraded").await;
let rollback = json!({"type": "rollback", "actor": "snapshot", "version": 1});
ask_type(&host, &mut mailbox, "broker", rollback, "rolled_back").await;
for (_, path, text) in files {
let read = json!({"type": "read", "path": path});
let reply = ask_type(&host, &mut mailbox, "snapshot", read, "content").await;
assert_eq!(reply.body["text"], text);
}
host.stop().await;
}
#[tokio::test] #[tokio::test]
async fn stop_finishes_when_actor_stdin_is_full() { async fn stop_finishes_when_actor_stdin_is_full() {
let world = create_world("full-stdin-stop"); let world = create_world("full-stdin-stop");
@@ -819,74 +752,6 @@ async fn shutdown_reply_and_publish_survive_stop_and_upgrade() {
host.stop().await; host.stop().await;
} }
#[tokio::test]
async fn rapid_remove_recreate_remove_keeps_actor_manageable() {
let world = create_world("rapid-remove");
let wasm = toolbox_wasm().clone();
let (host, mut mailbox) = start(&world).await;
let source = passing_source(&world, "source");
spawn_type(&host, &mut mailbox, "rapid", &wasm, &source).await;
let stop = json!({"type": "stop", "actor": "rapid"});
ask_type(&host, &mut mailbox, "broker", stop, "stopped").await;
while SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.subsec_millis()
>= 100
{
tokio::time::sleep(Duration::from_millis(10)).await;
}
let archive_second = epoch_second();
let remove = json!({"type": "remove", "actor": "rapid"});
let first = ask_type(&host, &mut mailbox, "broker", remove.clone(), "removed").await;
assert_eq!(
epoch_second(),
archive_second,
"first remove crossed an epoch-second boundary"
);
// Restore the stopped actor from its archive to avoid a candidate smoke test.
sausage::world::copy_tree(
first.body["archive"]
.as_str()
.map(std::path::Path::new)
.unwrap(),
&world.actor_dir("rapid"),
)
.unwrap();
let scan = json!({"type": "scan"});
ask_type(&host, &mut mailbox, "broker", scan.clone(), "scanned").await;
let second = ask_type(&host, &mut mailbox, "broker", remove.clone(), "removed").await;
assert_eq!(
epoch_second(),
archive_second,
"remove and recreate crossed an epoch-second boundary"
);
sausage::world::copy_tree(
second.body["archive"]
.as_str()
.map(std::path::Path::new)
.unwrap(),
&world.actor_dir("rapid"),
)
.unwrap();
ask_type(&host, &mut mailbox, "broker", scan, "scanned").await;
let start = json!({"type": "start", "actor": "rapid"});
ask_type(&host, &mut mailbox, "broker", start, "started").await;
let echo = ask_type(
&host,
&mut mailbox,
"rapid",
json!({"type": "echo", "value": "managed"}),
"echo",
)
.await;
assert_eq!(echo.body["value"], "managed");
host.stop().await;
}
#[tokio::test] #[tokio::test]
async fn close_commits_open_file_after_parent_directory_rename() { async fn close_commits_open_file_after_parent_directory_rename() {
let (_, host, mut mailbox) = filesystem_world("rename-open-file").await; let (_, host, mut mailbox) = filesystem_world("rename-open-file").await;
examples/bench/Cargo.toml
-9
View File
@@ -1,9 +0,0 @@
[package]
name = "bench"
version = "0.1.0"
edition = "2024"
[dependencies]
sausage-actor = { version = "0.1", path = "../../sdk" }
[workspace]
examples/bench/src/main.rs
-42
View File
@@ -1,42 +0,0 @@
//! Ping-pong with a target and report the round-trip times.
use sausage_actor::{json, messages, reply, send};
use std::time::Instant;
fn main() {
let mut inbox = messages();
while let Some(message) = inbox.next() {
if message.kind() != "bench" {
continue;
}
let target = message.body["target"].as_str().unwrap_or("echo");
let n = message.body["n"].as_u64().unwrap_or(1000);
if n == 0 {
let _ = reply(&message, json!({"type": "error", "reason": "n must be greater than 0"}));
continue;
}
let mut times = Vec::new();
let started = Instant::now();
for i in 0..n {
let t = Instant::now();
let id = send(target, json!({"type": "ping", "i": i}));
loop {
let Some(reply) = inbox.next() else { return };
if reply.reply_to.as_deref() == Some(id.as_str()) {
break;
}
}
times.push(t.elapsed().as_micros() as u64);
}
let total = started.elapsed().as_secs_f64();
times.sort();
let p = |q: f64| times[((times.len() as f64 - 1.0) * q) as usize];
let _ = reply(
&message,
json!({
"type": "bench_result", "n": n, "total_s": total,
"msgs_per_s": (n as f64 / total).round(),
"p50_us": p(0.5), "p90_us": p(0.9), "p99_us": p(0.99), "max_us": p(1.0)
}),
);
}
}
examples/bench/tests/zero.json
-9
View File
@@ -1,9 +0,0 @@
{
"description": "reject zero benchmark samples",
"steps": [
{
"send": {"type": "bench", "n": 0},
"expect": {"type": "error", "reason": "n must be greater than 0"}
}
]
}
examples/crasher/Cargo.lock
Generated
-7
View File
@@ -1,7 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "crasher"
version = "0.1.0"
examples/crasher/Cargo.toml
-8
View File
@@ -1,8 +0,0 @@
[package]
name = "crasher"
version = "0.1.0"
edition = "2024"
[dependencies]
[workspace]
examples/crasher/src/main.rs
-6
View File
@@ -1,6 +0,0 @@
//! A bad version: it starts, waits 3 seconds (long enough for the smoke
//! test), then panics. Used to test the automatic rollback.
fn main() {
std::thread::sleep(std::time::Duration::from_secs(3));
panic!("this version is broken");
}
examples/echo/Cargo.toml
-9
View File
@@ -1,9 +0,0 @@
[package]
name = "echo"
version = "0.1.0"
edition = "2024"
[dependencies]
sausage-actor = { version = "0.1", path = "../../sdk" }
[workspace]
examples/echo/src/main.rs
-7
View File
@@ -1,7 +0,0 @@
use sausage_actor::{json, messages, reply};
fn main() {
for message in messages() {
let _ = reply(&message, json!({"type": "echo", "got": message.body}));
}
}
examples/fetch/Cargo.toml
-10
View File
@@ -1,10 +0,0 @@
[package]
name = "fetch"
version = "0.1.0"
edition = "2024"
[dependencies]
sausage-actor = { version = "0.1", path = "../../sdk", features = ["async"] }
wstd = "0.6"
[workspace]
examples/fetch/src/main.rs
-85
View File
@@ -1,85 +0,0 @@
//! Fetch URLs over HTTP while answering pings.
use sausage_actor::{Inbox, Message, json, me, reply};
use wstd::http::{Body, Client, Request};
fn next_fetch_id() -> Result<u64, String> {
let previous = match std::fs::read_to_string("/data/fetch-sequence") {
Ok(value) => value.parse::<u64>().map_err(|e| e.to_string())?,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => existing_fetch_id()?,
Err(e) => return Err(e.to_string()),
};
let next = previous
.checked_add(1)
.ok_or_else(|| "fetch ID space exhausted".to_string())?;
std::fs::write("/data/fetch-sequence", next.to_string()).map_err(|e| e.to_string())?;
Ok(next)
}
fn existing_fetch_id() -> Result<u64, String> {
let entries = match std::fs::read_dir("/public/fetch") {
Ok(entries) => entries,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(0),
Err(e) => return Err(e.to_string()),
};
Ok(entries
.filter_map(Result::ok)
.filter_map(|entry| entry.file_name().to_str()?.parse().ok())
.max()
.unwrap_or(0))
}
async fn fetch(n: u64, message: Message) {
let url = message.body["url"].as_str().unwrap_or_default();
let started = wstd::time::Instant::now();
let result = async {
let request = Request::get(url)
.body(Body::empty())
.map_err(|e| e.to_string())?;
let response = Client::new()
.send(request)
.await
.map_err(|e| format!("{e:?}"))?;
let status = response.status().as_u16();
let bytes = response
.into_body()
.contents()
.await
.map_err(|e| format!("{e:?}"))?
.to_vec();
std::fs::create_dir_all("/public/fetch").map_err(|e| e.to_string())?;
let path = format!("/public/fetch/{n}");
std::fs::write(path, &bytes).map_err(|e| e.to_string())?;
Ok::<_, String>((status, bytes.len()))
}
.await;
let ms = started.elapsed().as_millis();
let body = match result {
Ok((status, size)) => {
let id = me();
let actor = if id.is_empty() { "fetch" } else { &id };
json!({"type": "fetched", "status": status, "bytes": size, "path": format!("/actor/{actor}/public/fetch/{n}"), "ms": ms})
}
Err(reason) => json!({"type": "error", "reason": reason, "ms": ms}),
};
let _ = reply(&message, body);
}
fn main() {
wstd::runtime::block_on(async {
let inbox = Inbox::start();
while let Some(message) = inbox.recv().await {
match message.kind() {
"fetch" => match next_fetch_id() {
Ok(n) => wstd::runtime::spawn(fetch(n, message)).detach(),
Err(reason) => {
let _ = reply(&message, json!({"type": "error", "reason": reason}));
}
}
"ping" => {
let _ = reply(&message, json!({"type": "pong"}));
}
_ => {}
}
}
});
}
examples/fstest/Cargo.toml
-9
View File
@@ -1,9 +0,0 @@
[package]
name = "fstest"
version = "0.1.0"
edition = "2024"
[dependencies]
sausage-actor = { version = "0.1", path = "../../sdk" }
[workspace]
examples/fstest/src/main.rs
-38
View File
@@ -1,38 +0,0 @@
//! Filesystem probe. Keeps files open across messages.
use sausage_actor::{Value, json, messages, reply};
use std::{collections::HashMap, io::Write};
fn main() {
let mut open = HashMap::<String, std::fs::File>::new();
for msg in messages() {
let b = &msg.body;
let path = b["path"].as_str().unwrap_or_default();
let r = |result: std::io::Result<Value>| result.unwrap_or_else(|e| json!({"type": "error", "error": e.to_string()}));
let response = match msg.kind() {
"write" => r(std::fs::write(path, b["text"].as_str().unwrap_or_default()).map(|_| json!({"type": "ok"}))),
"open" => r((|| {
let mut file = std::fs::File::create(path)?;
file.write_all(b["text"].as_str().unwrap_or_default().as_bytes())?;
open.insert(path.into(), file);
Ok(json!({"type": "ok", "open": true}))
})()),
"append" => r((|| {
let mut file = std::fs::OpenOptions::new().append(true).open(path)?;
file.write_all(b["text"].as_str().unwrap_or_default().as_bytes())?;
open.insert(path.into(), file);
Ok(json!({"type": "ok", "open": true}))
})()),
"close" => json!({"type": "ok", "closed": open.remove(path).is_some()}),
"read" => r(std::fs::read_to_string(path).map(|text| json!({"type": "content", "text": text}))),
"list" => r(std::fs::read_dir(path).map(|rd| {
let mut names: Vec<String> = rd.filter_map(|e| e.ok()).map(|e| e.file_name().to_string_lossy().to_string()).collect();
names.sort();
json!({"type": "entries", "names": names})
})),
"remove" => r(std::fs::remove_file(path).map(|_| json!({"type": "ok"}))),
"crash" => panic!("asked to crash"),
_ => json!({"type": "error", "error": "unknown message"}),
};
let _ = reply(&msg, response);
}
}
examples/notes-v2/Cargo.lock
Generated
+10
View File
@@ -18,6 +18,7 @@ checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
name = "notes" name = "notes"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"sausage-actor",
"serde_json", "serde_json",
] ]
@@ -39,6 +40,14 @@ dependencies = [
"proc-macro2", "proc-macro2",
] ]
[[package]]
name = "sausage-actor"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
]
[[package]] [[package]]
name = "serde" name = "serde"
version = "1.0.229" version = "1.0.229"
@@ -46,6 +55,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [ dependencies = [
"serde_core", "serde_core",
"serde_derive",
] ]
[[package]] [[package]]
examples/notes/Cargo.lock
Generated
+115
View File
@@ -0,0 +1,115 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "itoa"
version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
[[package]]
name = "memchr"
version = "2.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
[[package]]
name = "notes"
version = "0.1.0"
dependencies = [
"sausage-actor",
"serde_json",
]
[[package]]
name = "proc-macro2"
version = "1.0.107"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.47"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
dependencies = [
"proc-macro2",
]
[[package]]
name = "sausage-actor"
version = "0.1.0"
dependencies = [
"serde",
"serde_json",
]
[[package]]
name = "serde"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.229"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "serde_json"
version = "1.0.154"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e7e9cc8b1b85264074fbcc02a88680c4096b1e47df8f739dceb03bf482f04bd6"
dependencies = [
"itoa",
"memchr",
"serde",
"serde_core",
"zmij",
]
[[package]]
name = "syn"
version = "3.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d78c8dee4c7bf0e14673097256fed6142ce9d3b85a408189d07482442145823b"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "unicode-ident"
version = "1.0.27"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a2c754d6c33795a1c324727428e5a7dedb5b06195f9890bdbcba760d3e246563"
[[package]]
name = "zmij"
version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
examples/ticker/Cargo.toml
-9
View File
@@ -1,9 +0,0 @@
[package]
name = "ticker"
version = "0.1.0"
edition = "2024"
[dependencies]
sausage-actor = { version = "0.1", path = "../../sdk" }
[workspace]
examples/ticker/src/main.rs
-13
View File
@@ -1,13 +0,0 @@
//! Send 200 notes to `notes`, 50 ms apart, then stay alive.
use sausage_actor::{json, send};
use std::time::Duration;
fn main() {
for n in 1..=200 {
let _ = send("notes", json!({"type": "add", "text": format!("tick {n}")}));
std::thread::sleep(Duration::from_millis(50));
}
loop {
std::thread::sleep(Duration::from_secs(3600));
}
}