probierz benchmark pursue

Invocation#

probierz benchmark pursue --suite <SUITE> --case <CASE> --budget-usd <BUDGET_USD> <APP_ID>

Required inputs and options#

• The newest recorded run of `--suite` must lose `--case`. Our contender's program must live in a git checkout; that checkout is the pursuit's working directory. `--budget-usd` is the most the pursuit may spend on models and must be a positive amount.
• `jeden` on PATH, with the Brama and fleet-database settings `jeden pursue` needs.

Output and state effects#

Writes one durable Jeden request (`schema_version` 1) to `test-results/.benchmark/<app>/pursuits/<run-id>-<case>.json`: the objective names what the case asks, its input and assertions, who won it and by how much, what may not change (the suite file, any rival's driver), and the finish line — a new run whose standing shows ours winning or tying the case. Submits it to `jeden pursue` with write and command grants and reads Jeden's JSON response; an existing request file is submitted again unchanged, so Jeden answers a finished request from its own store and resumes nothing twice. When Jeden reports `succeeded`, Probierz records a new run of every declared contender, reads that run's verdict for the case, brings the catalog roadmap in line with it (`benchmark roadmap`, which records the item of a case ours now wins as delivered with that run's id as its decision), writes `<request>.outcome.json` beside the request, and prints the request, Jeden's response, the acceptance run and its standing, `accepted`, and the roadmap change.

Exact refusals#

• `--budget-usd <v> is not a positive amount of US dollars` and `suite <id> holds no case <case>; its cases are <ids>` before anything is written.
• Everything `benchmark standing` refuses, including `no run of suite <id> is recorded for <app>`, and `run <run-id> does not lose case <case> (its verdict is <verdict>); nothing to pursue`.
• `our contender <id> at <program> is not in a git checkout; a pursuit changes our product in its checkout`.
• `jeden could not start (<error>)`, naming the Stado command that installs Jeden (exit 1, prerequisite), and `jeden pursue exited <status> without a response document: <stderr>`.
• `Jeden's pursuit <request> ended <state>: <error>; case <case> stays lost`, naming the Jeden command that resumes a blocked request (refused), after printing Jeden's response; a response that names no state reads `answered without a state`, and one that names no error says `Jeden's response names no error` — neither is read as a state or an error of its own. The case is accepted only when the acceptance run's standing marks it `held` (ours among the case's best, alone or tied).
• `Jeden's pursuit <request> reported success, but run <run-id>, recorded after it, still shows case <case> <verdict>; the roadmap item stays` (refused), after printing the whole answer: an agent's own verdict never closes a lost case.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.