WisentProbierz
Probierz/Docs

CLI command

probierz benchmark author suite

Draft a suite for any product and its rivals, judge it, write it once and declare it.

Invocation#

Shell
probierz benchmark author suite --suite <SUITE> --cases <CASES> --rounds <ROUNDS> <APP_ID>

Required inputs and options#

  • The product's record and its rivals come from Stado's product catalog (the manifest's `productId`, else the app ID). `--cases` and `--rounds` are required and must be at least 1: how many cases the suite holds and how many drafts are allowed is the operator's decision.
  • The draft is made through the Stado model router the manifest or the environment names (`STADO_MODEL_ROUTER_URL`, `STADO_MODEL_ROUTER_TOKEN`, and for Brama `PROBIERZ_MODEL_AGENT_ID` and `PROBIERZ_MODEL_AGENT_SECRET`); Probierz holds no provider key.

Output and state effects#

The brief asks for deterministic cases on the promise the product's documentation makes, doable by every rival through its public interface, with host-specific values as `${PLACEHOLDER}` variables and expected values only in assertions. Each draft is written to `benchmark/<id>.json` in the manifest's first repository and judged by the same loader `benchmark run` uses; a refused draft is deleted and its refusal goes back to the model for the next round. An accepted suite is declared as `benchmark.suites.<id>` (with an empty `contenders` map if there was none). The answer carries the file, the suite hash, the case count, the variables and each round's model and usage.

Exact refusals#

  • `<file> already holds suite <id>; a suite changes under a new id or version, never in place, because runs compare only on the same suite hash`.
  • `after <n> round(s) no draft of suite <id> passed the suite loader; the last draft failed: <refusal>` (exit 1, refused).
  • A model router that is unset or refuses is named with its own sentence (exit 75, unavailable); a catalog without the product is refused as in `benchmark rivals`.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.