WisentProbierz
Probierz/Docs

CLI command

probierz benchmark author contender

Draft, place, declare and verify the driver of one contender, ours or a rival the catalog names, so no driver is written by hand.

Invocation#

Shell
probierz benchmark author contender --contender <CONTENDER> --suite <SUITE> --rounds <ROUNDS> <APP_ID>

Required inputs and options#

  • Without `--ours`, `<id>` must be a rival the catalog names for the product, and the brief carries its catalog record (name, URL, evidence). With `--ours` the brief carries the product's own record. `--suite` names a declared suite whose cases (never their assertions) go into the brief. `--rounds` is required: how many drafts are allowed.
  • The same model router settings as `benchmark author suite`.

Output and state effects#

The brief states the contender contract and the rules every driver follows: drive the product through its own published SDK, CLI or API; pass the case instruction and input unchanged; take every setting from a declared variable (packages from `<ID>_MODULES` or `<ID>_SITE_PACKAGES`); send model calls through Brama (`BRAMA_BASE_URL`, `BRAMA_API_KEY`, `BRAMA_MODEL`); no polling loops; one self-contained file. The model answers `{file, env, source}`. The driver is written executable to `benchmark/contenders/<id>/` (ours) or `benchmark/rivals/<id>/` (rivals; the only place a Python driver may live) in the manifest's first repository, replacing an earlier draft, and declared as `benchmark.contenders.<id>` with its `env`. It is then verified by a recorded `benchmark run` of that contender alone with one repetition. A broken attempt or one answered `failed` goes back to the model with the driver and the run's findings; a wrong answer is the measurement and is not repaired. A contender already declared is verified first and redrafted only if it fails, so the same command repairs a driver a rival's release broke. The answer lists every round with its run ID.

Exact refusals#

  • `the catalog names no rival <id> for <product>`, naming the Stado registry command that adds one, or `--ours` to author our own contender.
  • `no contender is ours yet, so no rival can be measured; probierz benchmark author contender <app> --contender <id> --ours --suite <id> drafts ours first` (exit 1, prerequisite), and `<id> is already our contender; one product has one`.
  • `<manifest> declares no suite <id>; probierz benchmark author suite <app> --suite <id> drafts one`.
  • `the declared <id> driver <path> cannot be read: <cause>` (exit 1, config) when the manifest declares a driver that exists but cannot be read; a missing driver file is drafted from nothing, an unreadable one is never drafted over.
  • A variable the drafted driver declares but this host does not set stops verification with `contender <id> declares variable <NAME>, which is not set in this environment`; the driver and its declaration stay, and the same command verifies them once the variable is set.
  • `after <n> round(s) the <id> driver still breaks the contract or fails: <findings>` (exit 1, refused), after printing every round.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.