CLI command

probierz benchmark survey

Build an existing product's whole benchmark from its name: its rivals, the suite's cases, a driver for every contender, a run and its standing.

Invocation#

Shell
probierz benchmark survey --suite <SUITE> --cases <CASES> --rounds <ROUNDS> --count <COUNT> <APP_ID>

Required inputs and options#

  • --suite names the suite: the one the manifest declares under that id, or a new one drafted under it. --cases, --rounds and --count are required and must be at least 1; they are passed to the authoring steps. --repetitions overrides the suite's for the run.
  • Everything the steps read: Stado's product catalog and inference alternative, and the variables each declared driver needs to run.

Output and state effects#

Runs, as this binary's own commands: benchmark author rivals when the catalog names no rival; benchmark author suite when the manifest declares no suite of that id, then names the suite in the catalog's benchmark of the product beside the ones it names (so benchmark cycle runs it); benchmark author contender --ours (contender id the app ID) when no contender is ours; benchmark author contender for every catalogued rival without a driver; benchmark run and benchmark standing. What an earlier survey made is not made again. A refused step every later step reads ends the survey; a rival whose driver cannot be drafted stays out of the run with its refusal in the report. Every step's arguments, exit, answer and refusal are kept in test-results/.survey/<app>/survey-<timestamp>.json (ai.wisent.probierz.benchmark.survey.v1); the answer names the report, each step and whether it held, the run's id and summary and the standing.

Exact refusals#

  • Usage (exit 2) of probierz.benchmark.survey: counts-zero for --cases, --rounds or --count of 0, before anything is read.
  • self-unnamed when the operating system cannot name this executable, which every step runs; unwritten when the report or its directory under test-results/.survey/ cannot be written.
  • stopped when the rivals, suite, our driver, run or standing step refused, quoting the step's own refusal (exit 1), after the report is kept and the answer printed.
  • A manifest that cannot be loaded, a catalog without the product, and the registry's refusal of the suite's catalog entry are refused by their own operations.

Source snapshot#

Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.

Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.