cargo.build: a persistent build cache shared by all worlds

- The cargo target folder is <cache>/cargo-target, next to Wasmtime's
  cache, which moves to <cache>/wasmtime: Wasmtime's cleanup worker
  deletes unknown entries in its folder, also Cargo's artifacts.
- Builds use the SDK at <cache>/sdk/<content hash>/, so cargo reuses
  the compiled SDK in every world. /plugins/cargo/sdk/ stays for agents.
- Tests run in the release profile and share compiled crates with the
  release build.
- The cargo and cli tests use a persistent cache in CARGO_TARGET_TMPDIR:
  warm, the cargo suite takes 11 s (was 235 s) and the cli suite 6 s
  (was 149 s).
- nix flake check runs ignored stress tests. AGENTS.md: rules for fast
  tests.

AGENTS.md

@@ -35,11 +35,17 @@ Keep it that way.
 ## Checks
 
 ```sh
-nix flake check -L      # all tests below; CI runs this on every push
-nix develop -c cargo test --workspace
+nix flake check -L      # everything below, plus the stress tests; CI runs this on every push
+nix develop -c cargo test --workspace     # fast with a warm cache: run it during development
 nix develop -c cargo build --release -p agent --target wasm32-wasip2
 nix develop -c cargo clippy --workspace
 ```
 
 `nix run -- init` and `nix run -- run` must keep working without other
 setup.
+
+Tests must stay fast. Only a few tests may wait for a timing (grace periods,
+deadlines). Poll for a condition with a deadline, do not sleep. `#[ignore]` is
+only for stress tests: `nix flake check` runs them with `--include-ignored`.
+Tests that build plugins use the persistent cache in
+`env!("CARGO_TARGET_TMPDIR")`.

DECISIONS.md

@@ -69,6 +69,12 @@ a plugin, or a change that the agent makes to itself.
   core.
 - **The SDK is not published.** `cargo` points `sausage-plugin = "0.1"` at
   its own copy, which it also shows in `/plugins/cargo/sdk/`.
+- **Compiled code is a cache, not state.** It lives outside the world, in
+  one cache folder that all worlds share: `wasmtime/` (owned by Wasmtime,
+  whose cleanup deletes unknown entries), `cargo-target/`, and
+  `sdk/<hash>/`. A path that depends only on the content of the SDK lets
+  cargo reuse the compiled SDK in every world. Tests run in the release
+  profile, so that they share compiled crates with the release build.
 - **Discovery has a deadline of 10 seconds.** It is an internal call of the
   core, not a limit on plugins. Without it, one plugin that does not answer
   would block every activation.

PROTOCOL.md

@@ -411,16 +411,19 @@ The core has one native plugin: `cargo`. `cargo.build {"plugin": "notes"}`
 builds `/plugins/notes/source/`:
 
 1. It copies the source once. The tests and the build both use this copy.
-2. It runs the tests under WASI.
-3. It builds the release `.wasm` for `wasm32-wasip2`.
+2. It runs the tests under WASI, in the release profile.
+3. It builds the release `.wasm` for `wasm32-wasip2`. The tests and the build
+   share the compiled dependencies.
 4. It returns `{"wasm": "/state/cargo/builds/<n>/notes.wasm", "tests":
    {"passed", "failed", "output"}, "warnings"}`. If a step fails, it returns
    an error with the compiler or test output.
 
 - A call to `cargo.build` gets a process ID, a folder in `/proc`, and journal
   entries, like other calls.
-- The dependency `sausage-plugin = "0.1"` points to the SDK copy at
-  `/plugins/cargo/sdk/`.
+- `/plugins/cargo/sdk/` shows the source of the SDK. The dependency
+  `sausage-plugin = "0.1"` points to the same files in the host's cache.
+- Compiled crates stay in the host's cache, outside the world, and all worlds
+  share them. So a second build compiles only what changed.
 - Build scripts can compile C: `CC_wasm32_wasip2` and `AR_wasm32_wasip2`
   point to a WASI clang.
 - The build does not activate the result. Call `core.activate` next.

README.md

@@ -17,8 +17,9 @@ an LLM (OpenAI Responses or Anthropic Messages API). No clone needed:
 nix run git+https://gitea.surma.technology/surma/sausage
 ```
 
-This creates a world in `./world` the first time (it builds the agent, which
-takes a few minutes), then starts it with a console. Later runs start the
+This creates a world in `./world` the first time (it builds the agent: the
+first build on a machine takes a few minutes, later worlds reuse the compiled
+crates), then starts it with a console. Later runs start the
 existing world. The two steps on their own:
 
 ```sh
@@ -32,6 +33,10 @@ the base URL, the model, and the token. It reads `SAUSAGE_API`,
 values go to `world/config/agent.json`. Both commands take another folder as
 an argument, for example `... -- run my-world`.
 
+Compiled code is cached outside the world, in `$XDG_CACHE_HOME/sausage/`
+(`~/.cache/sausage/` without the variable). All worlds share it.
+`--cache-dir <folder>` changes it for `init` and `run`.
+
 To keep a world on the newest version, run it in a loop:
 
 ```sh

cli/src/cargo.rs

@@ -3,6 +3,7 @@ use crate::{Host, world::World};
 use anyhow::{Context, Result, bail};
 use include_dir::{Dir, include_dir};
 use serde_json::{Value, json};
+use sha2::{Digest, Sha256};
 use std::{
     path::{Path, PathBuf},
     sync::Arc,
@@ -58,10 +59,12 @@ pub struct Cargo {
     world: World,
     builds: Arc<tokio::sync::Mutex<()>>,
     runner: PathBuf,
+    target: PathBuf,
+    sdk: PathBuf,
 }
 impl Cargo {
     /// Installs changed SDK files and selects the current sausage executable as runner.
-    pub fn new(world: World) -> Result<Self> {
+    pub fn new(world: World, cache: &Path) -> Result<Self> {
         let mut runner = std::env::current_exe()?;
         if runner
             .parent()
@@ -69,28 +72,29 @@ impl Cargo {
         {
             runner = runner.parent().unwrap().parent().unwrap().join("sausage");
         }
-        Self::with_runner(world, runner)
+        Self::with_runner(world, runner, cache)
     }
     /// Installs the SDK and accepts an explicit sausage binary, including for init integrations.
-    pub fn with_runner(world: World, runner: PathBuf) -> Result<Self> {
+    pub fn with_runner(world: World, runner: PathBuf, cache: &Path) -> Result<Self> {
         let sdk = world.plugin_dir(ID).join("sdk");
-        write_dir(&SDK, &sdk.join("src"))?;
-        write_dir(&MACROS, &sdk.join("macros/src"))?;
-        write_dir(&INTERFACE, &sdk.join("interface/src"))?;
+        let mut files = std::collections::BTreeMap::new();
+        collect_files(&SDK, Path::new("src"), &mut files);
+        collect_files(&MACROS, Path::new("macros/src"), &mut files);
+        collect_files(&INTERFACE, Path::new("interface/src"), &mut files);
         for (manifest, destination, interface_path) in [
             (
                 include_str!("../../sdk/Cargo.toml"),
-                sdk.join("Cargo.toml"),
+                PathBuf::from("Cargo.toml"),
                 Some("interface"),
             ),
             (
                 include_str!("../../sdk/macros/Cargo.toml"),
-                sdk.join("macros/Cargo.toml"),
+                PathBuf::from("macros/Cargo.toml"),
                 Some("../interface"),
             ),
             (
                 include_str!("../../interface/Cargo.toml"),
-                sdk.join("interface/Cargo.toml"),
+                PathBuf::from("interface/Cargo.toml"),
                 None,
             ),
         ] {
@@ -108,9 +112,40 @@ impl Cargo {
                     }
                 }
             }
-            write_changed(&destination, document.to_string().as_bytes())?;
+            files.insert(destination, document.to_string().into_bytes());
+        }
+        let mut hash = Sha256::new();
+        for (path, bytes) in &files {
+            let name = path.to_str().unwrap().as_bytes();
+            hash.update((name.len() as u64).to_le_bytes());
+            hash.update(name);
+            hash.update((bytes.len() as u64).to_le_bytes());
+            hash.update(bytes);
+            write_changed(&sdk.join(path), bytes)?;
+        }
+        let cached = std::path::absolute(cache)?
+            .join("sdk")
+            .join(format!("{:x}", hash.finalize()));
+        if !cached.is_dir() {
+            let parent = cached.parent().unwrap();
+            std::fs::create_dir_all(parent)
+                .with_context(|| format!("create {}", parent.display()))?;
+            let staging = tempfile::Builder::new()
+                .prefix("install-")
+                .tempdir_in(parent)?;
+            for (path, bytes) in &files {
+                write_changed(&staging.path().join(path), bytes)?;
+            }
+            // Another builder can publish the same complete, content-addressed SDK first.
+            if let Err(error) = std::fs::rename(staging.path(), &cached)
+                && !cached.is_dir()
+            {
+                return Err(error).with_context(|| format!("install {}", cached.display()));
+            }
         }
         Ok(Self {
+            sdk: cached,
+            target: std::path::absolute(cache)?.join("cargo-target"),
             world,
             runner,
             builds: Arc::new(tokio::sync::Mutex::new(())),
@@ -136,12 +171,12 @@ impl Cargo {
     pub async fn build(&self, source: &Path) -> std::result::Result<BuildOutput, BuildFailure> {
         let build_guard = self.builds.clone().lock_owned().await;
         let work = self.world.state_dir(ID).join("work");
-        let target = self.world.state_dir(ID).join("target");
+        let target = self.target.clone();
         let temporary = self.world.state_dir(ID).join("tmp");
         let cleanup = temporary.clone();
         let source = source.to_path_buf();
         let copy = work.clone();
-        let sdk = self.world.plugin_dir(ID).join("sdk");
+        let sdk = self.sdk.clone();
         let build_guard = tokio::task::spawn_blocking(move || -> Result<_> {
             if cleanup.exists() {
                 std::fs::remove_dir_all(&cleanup)
@@ -168,7 +203,7 @@ impl Cargo {
         .await
         .map_err(anyhow::Error::from)??;
         let mut test = Command::new("cargo");
-        test.args(["test", "--target", TARGET, "--target-dir"])
+        test.args(["test", "--release", "--target", TARGET, "--target-dir"])
             .arg(&target)
             .current_dir(&work)
             .env("TMPDIR", &temporary)
@@ -372,15 +407,18 @@ fn write_changed(path: &Path, bytes: &[u8]) -> Result<()> {
     }
     Ok(())
 }
-/// Installs one embedded source tree without touching unchanged files.
-fn write_dir(directory: &Dir<'_>, destination: &Path) -> Result<()> {
+/// Collects embedded source files for deterministic hashing and installation.
+fn collect_files(
+    directory: &Dir<'_>,
+    destination: &Path,
+    files: &mut std::collections::BTreeMap<PathBuf, Vec<u8>>,
+) {
     for file in directory.files() {
-        write_changed(&destination.join(file.path()), file.contents())?;
+        files.insert(destination.join(file.path()), file.contents().to_vec());
     }
     for directory in directory.dirs() {
-        write_dir(directory, destination)?;
+        collect_files(directory, destination, files);
     }
-    Ok(())
 }
 /// Counts only final libtest summaries, never individual status lines.
 fn test_count(output: &str, field: &str) -> u64 {

cli/src/host.rs

@@ -58,6 +58,7 @@ impl Host {
     /// Start a world with an explicit compile cache folder.
     pub async fn start_with_cache(path: &Path, cache: &Path) -> anyhow::Result<Self> {
         let path = path.to_path_buf();
+        let cargo_cache = cache.to_path_buf();
         let cache = cache.to_path_buf();
         let (world, runtime) = tokio::task::spawn_blocking(move || -> anyhow::Result<_> {
             Ok((World::open(&path)?, Runtime::new(&cache)?))
@@ -67,8 +68,10 @@ impl Host {
             journal::Journal::open(world.state_dir("core").join("journal.sqlite")).await?;
         let host = Self::with_parts(world, runtime, journal);
         let cargo_world = host.0.world.clone();
-        let cargo =
-            tokio::task::spawn_blocking(move || crate::cargo::Cargo::new(cargo_world)).await??;
+        let cargo = tokio::task::spawn_blocking(move || {
+            crate::cargo::Cargo::new(cargo_world, &cargo_cache)
+        })
+        .await??;
         cargo.register(&host);
         host.boot().await?;
         Ok(host)

cli/src/main.rs

@@ -79,7 +79,7 @@ fn setting(var: &str, question: &str, default: Option<&str>) -> Result<String> {
     }
 }
 /// Creates a configured world and builds its embedded agent through the standalone builder.
-async fn init(dir: &Path) -> Result<()> {
+async fn init(dir: &Path, cache: Option<&Path>) -> Result<()> {
     let api = setting("SAUSAGE_API", "API shape", Some("openai-responses"))?;
     if api != "openai-responses" && api != "anthropic-messages" {
         bail!("unknown API shape {api:?}");
@@ -105,7 +105,15 @@ async fn init(dir: &Path) -> Result<()> {
             &json!({"api":api,"base_url":base_url.trim_end_matches('/'),"model":model,"token":token}),
         )?,
     )?;
-    let builder = sausage::cargo::Cargo::new(world.clone())?;
+    let cache = cache.map(Path::to_path_buf).unwrap_or_else(|| {
+        let base = std::env::var_os("XDG_CACHE_HOME")
+            .map(PathBuf::from)
+            .unwrap_or_else(|| {
+                PathBuf::from(std::env::var_os("HOME").unwrap_or_default()).join(".cache")
+            });
+        base.join("sausage")
+    });
+    let builder = sausage::cargo::Cargo::new(world.clone(), &cache)?;
     let source = world.plugin_dir("agent").join("source");
     std::fs::create_dir_all(source.join("src"))?;
     AGENT.extract(source.join("src"))?;
@@ -133,7 +141,7 @@ async fn main() -> Result<std::process::ExitCode> {
         None => {
             let dir = PathBuf::from("world");
             if !dir.join("config/boot.json").is_file() {
-                init(&dir).await?;
+                init(&dir, cli.cache_dir.as_deref()).await?;
             }
             Cmd::Run { dir }
         }
@@ -144,7 +152,7 @@ async fn main() -> Result<std::process::ExitCode> {
                 return Ok(std::process::ExitCode::FAILURE);
             }
         }
-        Cmd::Init { dir } => init(&dir).await?,
+        Cmd::Init { dir } => init(&dir, cli.cache_dir.as_deref()).await?,
         Cmd::Run { dir } => {
             let mut term =
                 tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())?;

cli/src/runtime.rs

@@ -222,7 +222,8 @@ impl Runtime {
         let mut config = Config::new();
         config.epoch_interruption(true);
         let disk_cache = (|| -> Result<_> {
-            std::fs::create_dir_all(cache)?;
+            let cache = cache.join("wasmtime");
+            std::fs::create_dir_all(&cache)?;
             let mut cache_config = wasmtime::CacheConfig::new();
             cache_config.with_directory(cache.canonicalize()?);
             Ok(wasmtime::Cache::new(cache_config)?)

cli/tests/cargo.rs

@@ -3,6 +3,7 @@ use sausage::{Host, world::World};
 use serde_json::{Value, json};
 use std::path::{Path, PathBuf};
 use tokio::sync::OnceCell;
+const CACHE: &str = concat!(env!("CARGO_TARGET_TMPDIR"), "/cache");
 struct TestWorld(PathBuf);
 impl TestWorld {
     /// Creates a fresh native build world.
@@ -45,10 +46,12 @@ fn fixture(name: &str) -> PathBuf {
 }
 /// Builds through the real registered native plugin.
 async fn build(world: &TestWorld, name: &str) -> Result<Value, Value> {
-    let host = Host::start(&world.0).await.unwrap();
+    let host = Host::start_with_cache(&world.0, Path::new(CACHE))
+        .await
+        .unwrap();
     let runner = world.0.join("state/runner with spaces");
     std::os::unix::fs::symlink(env!("CARGO_BIN_EXE_sausage"), &runner).unwrap();
-    sausage::cargo::Cargo::with_runner(World::open(&world.0).unwrap(), runner)
+    sausage::cargo::Cargo::with_runner(World::open(&world.0).unwrap(), runner, Path::new(CACHE))
         .unwrap()
         .register(&host);
     let result = host.call("cargo.build", json!({"plugin":name})).await;
@@ -93,6 +96,7 @@ async fn valid() -> &'static Valid {
                 .args([
                     "test",
                     "--no-run",
+                    "--release",
                     "--message-format=json",
                     "--bin",
                     "cargo-valid",
@@ -100,7 +104,7 @@ async fn valid() -> &'static Valid {
                     "wasm32-wasip2",
                     "--target-dir",
                 ])
-                .arg(world.0.join("state/cargo/target"))
+                .arg(Path::new(CACHE).join("cargo-target"))
                 .current_dir(world.0.join("state/cargo/work"))
                 .stdin(std::process::Stdio::null())
                 .output()
@@ -221,7 +225,16 @@ async fn manifest_and_nested_sources() {
     let manifest = std::fs::read_to_string(root.join("Cargo.toml")).unwrap();
     assert!(manifest.contains("workspace.package"));
     assert!(manifest.contains("optional = true"));
-    assert!(manifest.contains("/plugins/cargo/sdk"));
+    let document: toml_edit::DocumentMut = manifest.parse().unwrap();
+    let sdk = Path::new(document["target"]["cfg(target_arch = \"wasm32\")"]["dependencies"]["sausage_plugin"]["path"].as_str().unwrap());
+    assert_eq!(sdk.parent().unwrap(), Path::new(CACHE).join("sdk"));
+    let hash = sdk.file_name().unwrap().to_str().unwrap();
+    assert_eq!(hash.len(), 64);
+    assert!(hash.bytes().all(|byte| byte.is_ascii_hexdigit()));
+    assert_eq!(
+        std::fs::read(sdk.join("src/lib.rs")).unwrap(),
+        std::fs::read(valid.world.0.join("plugins/cargo/sdk/src/lib.rs")).unwrap()
+    );
     assert_eq!(
         std::fs::read_to_string(root.join("src/target/data")).unwrap(),
         "nested target"
@@ -235,7 +248,9 @@ async fn manifest_and_nested_sources() {
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
 async fn native_discovery_and_error_paths() {
     let world = TestWorld::new();
-    let host = Host::start(&world.0).await.unwrap();
+    let host = Host::start_with_cache(&world.0, Path::new(CACHE))
+        .await
+        .unwrap();
     assert_eq!(
         host.call("cargo.rpc.discover", json!({})).await.unwrap()["methods"][0]["name"],
         "build"
@@ -264,7 +279,7 @@ async fn release_only_failure() {
         "[package]\nname='release-only'\nversion='0.1.0'\nedition='2024'\n",
     )
     .unwrap();
-    std::fs::write(root.join("main.rs"), "#[cfg(not(debug_assertions))] compile_error!(\"release-only error\"); fn main() {} #[test] fn passes() {}\n").unwrap();
+    std::fs::write(root.join("main.rs"), "#[cfg(not(test))] compile_error!(\"release-only error\"); fn main() {} #[test] fn passes() {}\n").unwrap();
     let error = build(&world, "release").await.unwrap_err();
     assert!(
         error["data"]["output"]
@@ -296,6 +311,7 @@ async fn cancelled_preparation_is_serialized() {
         sausage::cargo::Cargo::with_runner(
             World::open(&world.0).unwrap(),
             PathBuf::from(env!("CARGO_BIN_EXE_sausage")),
+            Path::new(CACHE),
         )
         .unwrap(),
     );
@@ -358,6 +374,12 @@ async fn valid_build() {
     assert_eq!(valid.result["tests"]["passed"], 12);
     assert_eq!(valid.result["tests"]["failed"], 0);
     assert!(!valid.world.0.join("plugins/valid/plugin.wasm").exists());
+    assert!(!valid.world.0.join("state/cargo/target").exists());
+    assert!(
+        Path::new(CACHE)
+            .join("cargo-target/wasm32-wasip2/release")
+            .is_dir()
+    );
 }
 /// PROTOCOL 11: tests and compilation share one source copy without target metadata.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
@@ -373,7 +395,7 @@ async fn source_copy() {
     assert!(
         std::fs::read_to_string(valid.world.0.join("state/cargo/work/Cargo.toml"))
             .unwrap()
-            .contains("/plugins/cargo/sdk")
+            .contains(Path::new(CACHE).join("sdk").to_str().unwrap())
     );
 }
 /// PROTOCOL 11: compiler failures return compiler diagnostics.
@@ -514,6 +536,7 @@ async fn unchanged_sdk() {
         sausage::cargo::Cargo::with_runner(
             World::open(&world.0).unwrap(),
             PathBuf::from(env!("CARGO_BIN_EXE_sausage")),
+            Path::new(CACHE),
         )
         .unwrap()
     };
@@ -534,6 +557,7 @@ async fn standalone_builder() {
     let builder = sausage::cargo::Cargo::with_runner(
         World::open(&world.0).unwrap(),
         PathBuf::from(env!("CARGO_BIN_EXE_sausage")),
+        Path::new(CACHE),
     )
     .unwrap();
     assert_eq!(
@@ -555,7 +579,9 @@ async fn kill_process_group() {
     );
     let world = TestWorld::new();
     world.source("hang", &fixture("cargo-hang"));
-    let host = Host::start(&world.0).await.unwrap();
+    let host = Host::start_with_cache(&world.0, Path::new(CACHE))
+        .await
+        .unwrap();
     let building = host.clone();
     let task =
         tokio::spawn(async move { building.call("cargo.build", json!({"plugin":"hang"})).await });

cli/tests/cli.rs

@@ -12,9 +12,15 @@ use tokio::{
     process::{Child, Command},
 };
 
+const CACHE: &str = concat!(env!("CARGO_TARGET_TMPDIR"), "/cache");
+
 fn binary() -> Command {
     let mut command = Command::new(env!("CARGO_BIN_EXE_sausage"));
-    command.env_remove("SAUSAGE_HEADLESS").kill_on_drop(true);
+    command
+        .arg("--cache-dir")
+        .arg(CACHE)
+        .env_remove("SAUSAGE_HEADLESS")
+        .kill_on_drop(true);
     command
 }
 fn world() -> (tempfile::TempDir, PathBuf) {
@@ -133,12 +139,15 @@ async fn init_builds_agent() {
         std::fs::read(world.join("plugins/agent/source/Cargo.lock")).unwrap(),
         include_bytes!("../../Cargo.lock")
     );
-    let host = Host::start_with_cache(
-        &world,
-        &Path::new(env!("CARGO_MANIFEST_DIR")).join("../target/core/cli-cache"),
-    )
-    .await
-    .unwrap();
+    assert!(!world.join("state/cargo/target").exists());
+    assert!(
+        Path::new(CACHE)
+            .join("cargo-target/wasm32-wasip2/release")
+            .is_dir()
+    );
+    let host = Host::start_with_cache(&world, Path::new(CACHE))
+        .await
+        .unwrap();
     let doc = host.call("agent.rpc.discover", json!({})).await.unwrap();
     assert_eq!(doc["x-sausage-grace"], 600);
     assert!(

flake.nix

@@ -98,6 +98,8 @@
         checks.tests = craneLib.cargoTest (common // wasiCcEnv // {
           inherit cargoArtifacts;
           cargoExtraArgs = "--locked ${testPackages}";
+          # Stress tests are ignored in a normal `cargo test`. The check runs them.
+          cargoTestExtraArgs = "-- --include-ignored";
           # Nothing builds on the test run's target folder: don't store it.
           doInstallCargoArtifacts = false;
           # The SDK tests run sample plugins with the wasmtime CLI.