WisentProbierz
Probierz/Docs

CLI command

probierz benchmark roadmap

Turn the newest run's lost cases into the product's catalog roadmap, and record the items of cases ours now wins as delivered with the run that won them.

Invocation#

Shell
probierz benchmark roadmap --suite <SUITE> <APP_ID>

Required inputs and options#

  • The newest recorded run of `--suite` is read through `benchmark standing`.

Output and state effects#

For every lost case it writes the roadmap item `Win benchmark case <suite>/<case>` (status planned, the outcome naming the standing that will prove it, the source naming the run) through the Stado registry; a case already open in the roadmap keeps its item, refreshed. An item of that suite still open, as the catalog's `roadmap_statuses` declares, whose case the newest run no longer loses stays in the roadmap with status delivered and that run's id as its decision, so the history of what the benchmark asked of the product reads back from `stado product catalog`. A delivered item is never written again: a case lost again after delivery is planned as the new item `Win benchmark case <suite>/<case>, lost again in run <run>`, and the delivered item keeps its decision. Both are written through the one roadmap writer the incident feedback of `benchmark cycle` uses, which then reads the catalog back and fails unless it holds every item with the status and decision written. Answers `{appId, product, runId, added, delivered}`.

Exact refusals#

  • Everything `benchmark standing` refuses, and the Stado registry's own refusal of the write (a Stado that keeps no roadmap history refuses status delivered).
  • `<product>'s catalog roadmap is <value>, not a list`; `the <suite> standing of <app> lists no losses`; `case <case> of the <suite> standing of <app> names no winners`.
  • Whether an item is still open is read from the `roadmap_statuses` (`{status, decided}`) the catalog answer declares, never from a list of Probierz's own: `stado product catalog --json declares no roadmap_statuses list (<value>) …`, `… declares the roadmap status <entry> without both status and decided`, and `roadmap item <title> has status <status>, which the catalog's roadmap_statuses do not declare`.
  • `the catalog read back after the write does not hold these items with the status written: <titles>`.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.