# End-to-end tests against real applications

Each subdirectory here runs a real application's *own* shell-integration test
suite inside a container, with the shell under test swapped in for `bash`.
This catches interactive-shell regressions (bindings, `bind -x`, `READLINE_*`,
completion, history) that unit tests can't.

```bash
cargo build
cargo xtask test e2e fzf                     # one adapter against brush
cargo xtask test e2e                         # default adapters (excludes blesh, bash-tests, bash-completion)
cargo xtask test e2e --with bash-completion  # default adapters plus an opt-in one
cargo xtask test e2e blesh                   # opt in to the slow ble.sh suite
cargo xtask test e2e --baseline fzf          # baseline against Bash
cargo xtask test e2e --shell /path/to/sh fzf # any other binary
cargo xtask test e2e fzf -- -n /ctrl_r/      # adapter-specific arguments
cargo xtask test --release e2e fzf           # use target/release/brush
cargo xtask test e2e --verbose fzf           # stream the runner's own output
cargo xtask test e2e --timeout 600 fzf      # bound each adapter (default 1800s)
cargo xtask test e2e --summary-output e2e.md # also write the run as a Markdown dashboard
```

A run reports one line per adapter and a summary, read back from the JUnit
each runner writes:

```
    Starting 7 e2e adapter(s) against brush
     Results target/e2e/run-ABC123
        PASS [    1.2s] starship  3 tests
        FAIL [   13.3s] atuin     5 tests, 1 failed
        PASS [  542.2s] blesh     9 tests, 9 xfailed
------------
     Summary [ 11m35.0s] 7 adapter(s): 6 passed, 1 failed

  atuin     failed: ctrl-r search runs the selected command
            -> target/e2e/run-ABC123/atuin/log.txt
```

Results land in `target/e2e/run-<id>/<app>/`: `log.txt` (full runner output),
`junit/*.xml` (JUnit, for CI upload), and any adapter-specific diagnostic logs
(such as PTY transcripts). Each invocation creates and prints a fresh run directory,
retaining previous runs and leaving existing files untouched, including when using
`--results-dir`. Missing, malformed, or empty JUnit reports fail the adapter, as does an adapter that runs longer
than `--timeout` (default 1800s): its container is killed, so a hung shell cannot hang the run.
The runner's own output goes to `log.txt` rather than the terminal; `--verbose`
streams it instead.
Multi-adapter runs continue after failures and report them together at the end.
On a terminal, statuses are green for clean passes, yellow for passes with expected
failures, and red for failures. Redirected output, `TERM=dumb`, and a nonempty
`NO_COLOR` disable color.

They need Docker and a built shell binary. CI runs the default selection on every
PR (the `e2e-tests` job in `ci.yaml`, against the release binary the build job
produced) and posts the `--summary-output` dashboard to the PR comment, alongside
the coverage and benchmark reports, with each failing adapter's `log.txt` in the
job log and every run directory in the `e2e-logs` artifact. Because the `xfail-list.txt`
files are the committed baseline, a PR that fixes or breaks a test shows it in its own
diff; no run against `main` is needed. `blesh` is excluded from the default selection
because compatibility failures and timeouts make it take several minutes against
brush. Run it explicitly with `cargo xtask test e2e blesh`, or select one file with
`cargo xtask test e2e blesh -- util`. Its per-file timeout remains 180 seconds.
`bash-tests` is likewise opt-in, until it has shown itself reliable. Neither runs in CI.
`bash-completion` is opt-in because its ~2000 tests take several minutes; CI adds it with
`--with bash-completion`, and it is report-only (below), so it cannot fail the job on test results.

## Adapter contract

An adapter is a directory `e2e/<app>/` containing a `Dockerfile` (build
context is `e2e/`, so it can `COPY shim /e2e/bin`) whose entrypoint:

1. **Receives the shell** via `$SHELL_UNDER_TEST`, an absolute path to a
   binary bind-mounted into the container (`/shell/<name>`). It's built on
   the host, so the image's glibc must be at least the host's. Most apps
   hard-code `bash`, so the shared `shim/bash` exec's `$SHELL_UNDER_TEST`;
   put `/e2e/bin` first on `PATH`. When the variable is unset the shim
   runs the real `bash`, which gives the baseline run. If the suite has its
   own knob for which shell to test (atuin's `ATUIN_TEST_BASH`), set that
   instead and skip the shim; a shim on `PATH` also captures test tooling
   written in bash (tmux, say), which must not run under the shell under test.
   A suite that takes a shell *path* has nowhere to put a command-line flag; turn
   the behavior on through brush's config file (`$HOME/.config/brush/config.toml`,
   and `HOME` is `/tmp` in the container) instead, which bash ignores and so leaves
   the baseline run alone.
2. **Writes results** to `/results`: `log.txt` plus JUnit XML under
   `junit/`. Write whatever else helps debugging there too.
3. **Exits non-zero** when any test fails. Entrypoints pipe the runner through
   `tee`, so exit `${PIPESTATUS[0]}` rather than `$?`, which is `tee`'s status.
   Known failures are not the adapter's problem: the runner reads them from
   `xfail-list.txt` (below) and decides the result from the JUnit, so an adapter that
   exits non-zero purely because of expected failures still passes.

The container runs as the host user with `HOME=/tmp`, so `chmod` anything
the tests write into.

## Reproducing a failure

The dashboard names each failing test with the command that runs it alone, built from the
adapter's `select-args.txt`: the adapter arguments that select one test, with `{test}` standing
for its name (fzf: `-n /{test}/`). Adapters without the file are assumed to take pytest's
`-k {test}`, since that is what every adapter's own tests use.

## Tracking known failures

Every dashboard row shows the pass rate: passed tests over those that ran (skips excluded, xfails
included).

An adapter containing a `report-only` file (a comment saying why) is tracked by that rate alone:
its failing tests are listed, collapsed, and its row is marked 📊, but they do not fail the
adapter; nor does the non-zero exit they cause. A run that produced no results still fails it: an
image that does not build, a timeout, or missing JUnit. Such a suite needs no `xfail-list.txt`;
the runner's own expected failures (`@pytest.mark.xfail`) are counted as xfailed as usual.
Everything below applies to the other adapters.

Two lists per adapter, both optional, both `#`-commented, one test name per line.

`xfail-list.txt` is the default home for anything that fails today. The test still **runs**;
the runner matches it against the JUnit and counts it as `xfailed` instead of a failure.
Crucially it also fails the run when such a test **passes** ("passed unexpectedly") or when an
entry **matches no test that ran** ("stale xfail entry" -- renamed, removed, or moved to
`skip-list.txt`), so a gap that closes is reported rather than absorbed, and the list cannot rot.
Stale detection needs the whole suite, so it is skipped when adapter arguments select a subset. Because the matching happens over JUnit, one mechanism
covers every runner. Name a test by the `name` its JUnit carries, or as `classname::name`.
These expectations apply only to brush runs; `--baseline` and `--shell` runs use
no brush expectations and fail on any reported test failure.

`skip-list.txt` is for tests that must **not** run: they hang, or they cannot exercise the shell
at all (nvm has one with a `/bin/zsh` shebang). This is the exception -- prefer `xfail-list.txt`,
because a skipped test reports nothing when it starts working -- and listing one in both files
fails the run as a stale xfail entry. The runner reports skipped tests by name, so a suite that
starts skipping something itself does not pass unnoticed.

Applying the list is mostly handled for you: `lib/run_pytest.sh` exports the file as
`$E2E_SKIP_LIST` and loads `lib/e2e_skip.py`, a pytest plugin that skips any test named in it.
Adapters that must decide before collection (blesh) or at image build time (nvm, which takes the
execute bit off) parse `$E2E_SKIP_LIST` themselves; fzf turns it into a minitest `--exclude`
regex, since its runner is upstream's.

Every entry in either list needs a comment saying why, with an issue link where one exists.

## Shared adapter code

Every adapter's own tests are pytest, so `lib/` (copied to `/e2e/lib`, on `PYTHONPATH`) carries
what they would otherwise each reinvent:

| | |
|-----|-------|
| `run_pytest.sh` | the pytest half of the contract: JUnit, log, skip list, exit status. Source it and call `e2e_run_pytest <report name>` |
| `rcfile.py` | `write_rc()`: a known prompt and no `HISTFILE`, then the app's own integration lines |
| `pty_shell.py` | drives a shell over a raw PTY (pexpect) and matches its output stream. The default |
| `tmux_shell.py` | drives a shell inside tmux (via `libtmux`) and matches the *rendered screen*. For full-screen TUIs, whose redraws a byte stream cannot answer questions about |
| `e2e_skip.py` | the skip-list pytest plugin, plus `parse()` for adapters that must skip before collection |

## Rules

1. **Never copy upstream files into this tree.** Clone the application at
   build time, pinned to a commit (not a branch).
2. **The Dockerfile is our own environment recipe.** Deriving its package
   list from upstream's Dockerfile or CI config is fine; say so in a comment,
   with the source and license.
3. **The entrypoint adapts upstream's runner to the results contract.** If the
   runner can't emit JUnit itself, add a reporter there (fzf: `minitest-ci`).
4. **Not every app has a suite to borrow.** When upstream only tests its own
   code (atuin, say), the adapter brings its own tests: a small terminal-driven
   suite that installs the app's hooks into the shell under test and checks the
   observable behavior. The contract is the same; only the author differs.

## Adapters

| app | notes |
|-----|-------|
| fzf | upstream `test/test_shell_integration.rb`, `TestBash` only; minitest + tmux, JUnit via `minitest-ci` |
| atuin | our own suite in `atuin/tests/` (pytest + tmux), written to be upstreamable. atuin's bash integration is bash-preexec, so the adapter turns on brush's `zsh-hooks` through a config file to exercise the native hooks rather than the `DEBUG`-trap emulation |
| bash-tests | bash's own regression suite (`tests/run-*`), from a bash-5.3 build that also provides its helper binaries and the `--baseline` shell; one pytest case per `run-*` script, which is sourced verbatim, on a terminal, with `diff` replaced by a function that captures the output for byte-exact comparison against the `.right` file. `PLATFORM_DIFFS` in `tests/test_bash.py` excuses the few differences bash itself shows on Linux. Failures report how many expected lines match, and `log.txt` ends with the total across the run. Args go to pytest: `-- -k alias`, `-- --subset minimal` for the scripts `run-minimal` runs, `-- --script-timeout 60` (default 30s per script) |
| bash-completion | upstream's `test/t` pytest suite (tag 2.18.0), in upstream's own rolling `fedoradev` test image, which has the commands under test installed; built with `make` as upstream's CI does, and run under `xvfb-run`. Report-only: upstream's `xfail` markers are its only expectations. Args go to pytest: `-- -k test_ls` |
| blesh | upstream `ble.sh --test`, one pytest case per test section with a process-group timeout (`BLESH_TEST_TIMEOUT`, default 180s); sections are read from ble.sh's own build output, so a section added upstream runs rather than being missed; full output and leftover per-section artifacts are retained; args select sections: `cargo xtask test e2e blesh -- util`. Every section is currently expected to fail: ble.sh gets far enough to emit no section summary at all |
| mise | selected upstream bash activation tests (pytest), run directly against the shell under test |
| nvm | upstream `test/fast/Listing versions` suite (urchin) plus interactive regression coverage for [#1173](https://github.com/reubeno/brush/issues/1173) |
| oh-my-git | our own suite in `oh-my-git/tests/` (pytest + direct PTY); upstream has no tests or releases, so the scripts are cloned at a pinned commit |
| starship | our own suite in `starship/tests/` (pytest + direct PTY) |
| zoxide | our own suite in `zoxide/tests/` (pytest + direct PTY) |