probierz benchmark run

Invocation#

probierz benchmark run --suite <SUITE> <APP_ID>

Required inputs and options#

• `--suite` names a declared suite. `--contender` chooses contenders; without it every declared contender runs.
• `--repetitions N` overrides the suite's repetitions and must be a positive number.

Output and state effects#

Before any case runs, every chosen contender is checked: its program must be an executable file and every variable its `env` names must be set. Cases then run case by case, repetition by repetition, contender by contender, each until its process ends; one progress line per sample goes to stderr. A contender that exits non-zero or prints something other than a result document is recorded as a `broken` sample with the reason, and the run continues. The run is written owner-only to `test-results/.benchmark/<app>/<run-id>.json` (`ai.wisent.probierz.benchmark.run.v1`) with the suite hash, each contender's program, arguments, variable names and git source revision, every sample, and the per-contender summary. The answer carries the run ID, the summary and the standing.

Exact refusals#

• `no suite <id> is declared; declared suites: <ids>`.
• `no contender <id> is declared; declared contenders: <ids>` and `contender <id> is named twice; name each contender once`.
• `contender <id> program <path> cannot be read (<error>); install it or correct benchmark.contenders.<id>.program`, `contender <id> program <path> is not an executable file`, `contender <id> declares variable <NAME>, which is not set in this environment`, and `suite <id> fills ${<PLACEHOLDER>} from <NAME>, which is not set in this environment` (exit 1, prerequisite).
• Suite files are refused with the file named: `schema is <s>, expected ai.wisent.probierz.benchmark.suite.v1`, `declares id <x>, but the manifest names it suite <id>`, `version is empty; a suite is compared only within one version`, `repetitions is 0; a case must run at least once`, `declares no case`, `variables.<P> = <V>: placeholders and variable names are UPPER_SNAKE_CASE`, `case <id> is declared twice`, `case <id> has an empty instruction`, `case <id> has no assertion, so no answer could fail it`, `case <id> pointer <p> is not a JSON Pointer; it must be empty or start with /`, and `case <id> assertion at <p> must set exactly one of equals, exists, includes`.
• A manifest without a benchmark is refused with the command that starts one: `declares no benchmark; probierz benchmark author suite <app> --suite <id> starts one`; an empty `suites` with `benchmark.suites declares no suite; probierz benchmark author suite <app> --suite <id> drafts one`, and an empty `contenders` with `benchmark.contenders declares no contender; probierz benchmark author contender <app> --contender <id> --ours --suite <id> drafts ours`.
• `run <run-id> is already recorded at <path>; a run is never rewritten`.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.