CLI command

probierz benchmark board

Write the public leaderboard of every market across products' newest runs.

Invocation#

Shell
probierz benchmark board --out <OUT>

Required inputs and options#

  • --app names a product; repeat it for several. Without it every product with a directory under test-results/.benchmark/ is on the board. --out is the directory written.

Output and state effects#

For each product, the newest run of each suite is scored on the axes of its catalog product, and every market it measures is one entry of that market: board.json (ai.wisent.probierz.benchmark.board.v1: {schema, generatedAt, apps, markets: {<market>: [{app, product, suite, runId, startedAt, ranking, axes, ours, definitions, unmeasured, scored}]}}) and index.html, a static page with no script and its style inline: per market, contenders by composite with their success rate and each axis's median and score, each axis linking the page its scale comes from. Answers the two paths, the apps and the markets.

Exact refusals#

  • Everything the manifest's declaration and benchmark standing's markets refuse for a product named.
  • probierz.benchmark.board: apps-unlistable when the runs directory cannot be listed, and board-unwritten when --out or a file in it cannot be written.

Source snapshot#

Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.

Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.