WisentProbierz
Probierz/Docs

CLI command

probierz benchmark cycle

One pass of the whole benchmark loop without the operator, under the policy he wrote once: new markets watched, new products scouted, judged and created, every suite run, losses pursued and failures turned into roadmap work.

Invocation#

Shell
probierz benchmark cycle

Required inputs and options#

  • `--policy` names the policy; without it, `autonomy.yaml` in the harness root. The policy holds `schemaVersion: 1`, `owner` (the GitHub owner new repositories are created under), `productsPerWeek`, `observations`, `rounds`, `cases`, `pursuitBudgetUsd` and `pursuitsPerProduct`; any other key is refused.
  • `trends` and `competitors` on PATH: both are CLI products in Stado's catalog that Probierz's CLI depends on, so Stado's product install of each CLI surface puts them there. Also Stado, Jeden for pursuits, and the model router settings `benchmark author suite` needs.

Output and state effects#

Runs `trends init` first when Trends answers `state_missing`. Gives every catalog product Trends does not watch yet a topic named after it, with the terms the model drafts from the product's record and one source of every query kind Trends already uses. Runs `trends ingest`. With `evidenceDir` set, none of that runs: the topics are the directory's `<topic>.json` files, in name order, and each `probierz benchmark scout` is given `--evidence <dir>/<topic>.json`; the report records the directory under `topics`. For every watched topic, while fewer products than `productsPerWeek` were created in the last seven days, runs `benchmark scout`, asks the model to judge the brief against written rules (evidence floor, real rivals shown by observations, a gap a benchmark case could show, no duplicate of the catalog) and runs `benchmark adopt` on an accepted brief, recording the product, the brief, the policy's SHA-256 and the verdict in `test-results/.autonomy/adoptions.jsonl`. For every suite the catalog names, runs `benchmark run`, `roadmap` and `standing`, and hands up to `pursuitsPerProduct` of the newest losses to `benchmark pursue` with `pursuitBudgetUsd`. Then every open incident in Probierz's failure register becomes a roadmap item `Repair incident <id>` of the catalog product its envelope names as the service (its outcome built from the incident's claim), written through the same roadmap writer as `benchmark roadmap`. An open item whose incident the register holds resolved stays in the roadmap with status delivered and the decision `probierz incident show <id> reads <state> at <resolved_at>`. A delivered item is never written again: an incident reopened after delivery is planned as the new item `Repair incident <id>, reopened <reopened_at>` from its latest reopening in the register. An incident or item the register cannot name is refused alone, naming it, in the product's `refused` list, and the rest of the product's change is still written: an open incident without an id or a claim, an item whose incident `probierz incident list --state all` does not list or does not hold resolved with its time (the item stays open), and an incident open again after delivery whose latest recorded reopening names no new item. Every step is the command an operator would type; its answer or refusal is kept in `test-results/.autonomy/cycle-<timestamp>.json`, and a refused step does not stop the next one. The answer names the report and counts only what succeeded — `topicsAdded`, `scouted`, `adopted`, `suitesMeasured` — next to `topicsRefused` and `suitesRefused`, so a cycle whose steps were all refused does not read as work done; the report says why each was refused.

Exact refusals#

  • `the loop acts only under a written policy (…), and <file> cannot be read: <reason>`, naming every key the policy holds, before anything runs.
  • `<file> is not a policy: <reason>` for an unknown or missing key, `schemaVersion must be 1`, `observations, rounds and cases must be at least 1`, and `pursuitBudgetUsd <v> is not a positive amount of US dollars`.
  • `cannot read the adoption ledger <path>, so the weekly limit cannot be counted: <cause>` when `test-results/.autonomy/adoptions.jsonl` exists and cannot be read; a missing ledger counts no adoptions.
  • A catalog Stado cannot serve is refused as in `benchmark rivals`. Refusals of the steps themselves are recorded in the report, not raised.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.