CLI command
probierz benchmark
Enter the benchmark group: run our product and its rivals on the same versioned cases, read the recorded runs, and keep the catalog's rivals and roadmap honest.
Invocation#
probierz benchmark <COMMAND>Required inputs and options#
- The app manifest declares `benchmark.suites.<id>: <suite file>` (absolute after `~/` expansion, or relative to the manifest directory) and `benchmark.contenders.<id>: {program, args, env, ours}`. Exactly one contender declares `ours: true`; an unknown key is refused by name.
Output and state effects#
A suite (`ai.wisent.probierz.benchmark.suite.v1`) holds cases, each with an instruction, an input and JSON Pointer assertions (`equals`, `exists` or `includes`), and optional `variables` mapping each `${NAME}` placeholder in a case input to the environment variable Probierz fills it from before a contender reads the task. A contender is a program that reads one `ai.wisent.probierz.benchmark.task.v1` document on stdin and writes one `ai.wisent.probierz.benchmark.result.v1` document on stdout: `status` completed or failed, the `output` the assertions read, and optional `steps`, `tokens`, `costUsd` and `error`. Its environment is empty except for the variables its `env` names. There is no composite score: a contender is described by its pass rate and its p50, p95 and p99 durations.
Exact refusals#
- A missing or unknown leaf prints the group usage and exits 2.
Source snapshot#
Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.
Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.