CLI command

probierz report

One report for every product: what is broken, since which change, with its evidence and how to reproduce it, and every measure of the plan against the operator's target.

Invocation#

Shell
probierz report [APP] [--coverage] [--freshness] [--detection] [--feedback] [--escapes] [--mutation] [--triage] [--oracles] [--latency] [--repairs] [--cost] [--truth] [--save] [--send]

Required inputs and options#

  • [APP] limits it to one product; every product with a manifest when omitted.
  • A section flag reads only that measure; with none every section and measure is read. --save keeps the report under test-results/.reports/; --send pages its summary through Stado's alert channels. The global --text prints it for a person.
  • --truth reads whether Probierz's own truth tests (.github/workflows/rust-gates.yml: gate truth, the CLI's parity with MCP and the service, the report) passed in their newest completed run on main: value 1 when GitHub says it succeeded, 0 when it ended otherwise, with the run's conclusion, headSha, url and createdAt, as gh run list answers in Probierz's own checkout (.probierz/probierz.yaml's repository), held against target.truth. A workspace with no Probierz checkout, or a GitHub that cannot answer, makes it a gap naming why, never 0. Real test: tests/report/sections.rs.

Output and state effects#

Sections: journeys (each journey's newest result, its age against cadence, the revision it broke at, its evidence, class, defect, bisect and cells, unverified when it passed but an artifact it names is missing or changed (a failed result stays failed and counted broken, naming the artifact), and flaky with the revision, the cell (target, device and conditions) and a passing and a failing run when one commit gave it both verdicts in one cell; a pass in one cell and a failure in another (another language, another surface) is the product failing in that cell, never flaky; a run's revision is a suite run's source.revision or a surface run's primary repository gitSha, and a run of changes not committed is at no commit and left out of flaky, since its commit is not what it measured; the summary proposes probierz bisect <app> <journey> --quarantine-until <date> for a flaky journey not held out, and a quarantine past its date no longer holds), schedules, benchmark (per product, each suite's newest recorded run: each case won, tied, lost or absent from, with the contenders that won it; a product whose recorded runs the report cannot read is its own entry with the cause, naming the file, beside the others' standings), matrix (declared cells with and without a result) and firstUse (each first use beside real users' funnel from the funnel operation of Stado's telemetry family). The summary says red journeys of one product that broke at one revision with one class once, as one incident naming each journey with its evidence; a red journey whose breaking revision is unknown shares nothing known and stands alone. Measures: coverage, freshness, detection, feedback, escapes, mutation, triage, oracles, latency, repairs, cost and truth, each with its value, target.<measure> and the value in the newest kept report; a measure below its target that got worse opens a defect of Probierz. Detection is the p50 of seconds from a breaking commit to its journey's first red run over every red episode the history keeps, ended or not (each first red run after a green one, its later red runs part of it): the commit bisect found when bisect's bad end is a red run of the episode, else the revision of its first red run; an episode whose commit the checkout does not hold is named in gap, never counted as zero. Repairs is the share of failed runs whose repair (test-results/<app>/repairs/<run>/result.json) turned green in its verification and whose commit is on the branch main tracks; a merged repair that never turned green is mergedNotGreen, never accepted. Coverage gives, beside the catalog products and the inventory's routes, traffic: what real users used, from Echo through the usage operation of Stado's telemetry family (each item a route, METHOD /path or tool verb with its count of uses; Probierz's own marked traffic is left out), as the share of uses whose item some journey's covers claims, held against target.coverage-traffic, with each product's uncovered items most used first; the summary names each product's most used uncovered item and the journey declaration that closes it. A product whose usage is not answered makes traffic a named gap, never a share of part of it. Latency holds each journey's declared budgets against what its runs measured: the latency entries of its report, the latency.json its evidence keeps — a web journey program's Core Web Vitals (lcp and inp in milliseconds and cls as its unitless score, each read by the one expression probierz author spec gives every web program, as web.dev defines them; a vital the browser does not record is left out, never zero), and the native harness's launch (Android's cold start, iOS's time to the foreground) and Android's janky-frames share — and the requests Probierz times itself on the api surface. Each is budgeted like any measure, for example latency: [{name: lcp, budgets: {default: {p50Ms: 2500, p95Ms: 4000}}}] with web.dev's good and poor LCP limits; a cls or janky-frames budget is a score or share in the same fields. Each sample keeps the revision of the product it measured (the run manifest's primary repository gitSha, else the head of the product's checkout when the run was kept; null when neither is known), and each measure gives revisions (per revision, oldest first, its p50Ms, count and firstAt) and change: the newest revision against the one before (from, to, p50MsBefore, p50MsAfter and ratio), null with fewer than two, so a page that got heavier or slower in a release says by how much. Real test: tests/latency/revisions.rs. Escapes are the errors users met — Echo's error groups through the error-groups operation of Stado's telemetry family, defects people filed through the reported operation of its defects family, for a product whose manifest names its App Store page as inventory.appStore, each review of one or two stars on the newest page of its reviews (source app-store: the title its message, the words its summary, met when written, on no catalog surface), and, for a product whose manifest names its support mailbox as inventory.supportMailbox (organization in the mail alternative, the address users write to, and how many days back it is read), each message to that address in those days, read through the message-list operation of Stado's mail family (source support-mailbox: the subject its message, the text its summary, met when received) — that no open Probierz defect of that product (of the journey the error names, else of a journey on the surface it was met on, else any) preceded; each app lists them beside the caught ones, and a gaps schedule writes a journey for each escape. A page whose reviews cannot be read (a Google Play link, a value that is no link, the feed not answering) is a gap named app-store; a support mailbox naming no organization, address or days, or one the mail family does not read, is a gap named support-mailbox. Real test: tests/report/measures/escapes.rs. Cost counts every model call Probierz makes (authoring, repair, oracles, security, SEO, talk and figure evaluations; those for no product under fleet) and the host time of runs; instance is the whole instance's spend, which budget.host-seconds-per-day and budget.model-usd-per-day are held against, whichever products the report covers. Its perDefect is what finding one product defect cost: everything the products ever spent, model dollars and host seconds, divided by the product defects Probierz found (each opening and reopening of a defect with an occurrence classed as the product's fault; a flaky, harness or environment red finds nothing), the model dollars held against target.cost-per-defect; with none found it is a gap naming what was spent, and with unpriced model answers it is known only from below. Truth is whether Probierz's own truth tests passed in their newest completed CI run on main, as --truth above reads it. A measure no Stado alternative serves yet is a named gap, not zero.

Exact refusals#

  • incomplete (exit 69, worth retrying): a section whose source cannot be read, the kept reports a trend is read from included, is printed with its cause in place of a value and the command names every one; no section is shown as empty or green. Each product of the journeys, benchmark, matrix and firstUse sections is read on its own: one that cannot be read is {appId, unreadable} with its cause beside every other product (incomplete names it <section>.<app>, the summary says its section could not be read, --text prints its journeys unreadable). Without [APP] the report covers every product discovery finds, one whose manifest cannot be read too, and each measure that reads that manifest is printed with its failure as the cause, instead of the whole report refusing.
  • save-failed (exit 1, naming the file) and send-failed (exit 69, with Stado's sentence): the report is still printed whole, then the command fails naming the delivery that did not happen; with both, or with unread sections too, one failure names each. Real test: tests/report/sections.rs and tests/report/benchmark.rs.

Declared refusals#

What probierz refusals list declares for this command at revision d2cd6156e8b8dc6204dab7b46ec0b34b2bb03cea (probierz 0.1.7 (revision unknown, not built from a git checkout)): each cause's id, the operation declaring it, its exit status and code, and its remedy.

  • harness-stale (probierz.receipt.findings, exit 1, refused): signing the receipt of {app} {release}: expected harness source is stale relative to the current Probierz checkout: the receipt expects {expected}, the checkout is {current}. sign with the harness SHA-256 probierz identity {app} prints, or check out the harness the runs were made with.
  • source-stale (probierz.receipt.findings, exit 1, refused): signing the receipt of {app} {release}: expected app source is stale relative to the current product checkout: the receipt expects {expected}, the checkout is {current}. sign with --source-sha the app source SHA-256 probierz identity {app} prints, or check out the source the runs were made at.
  • target-unstated (probierz.receipt.findings, exit 1, refused): signing the receipt of {app} {release}: {run} records build {build} and no target, so which surface that build is released on cannot be told. record {run} again: every run Probierz records names the target it ran.
  • record-unplaced (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: the run record names no manifest file in a directory, so its artifacts cannot be found beside it. read the run through probierz history {app}, which names every run's manifest file.
  • bundle-unrecorded (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: the protected artifact bundle is not recorded: the run removed its plaintext and names no protection.file. protect the run again with probierz protect {app} {run} <kind> --remove-source, which records its bundle.
  • bundle-misplaced (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: protected artifact is missing or escapes its product root: {bundle} {why}. put the bundle back below {root}, or protect the run again with probierz protect {app} {run} <kind> --remove-source.
  • bundle-hash-other (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: protected artifact hash mismatch: {bundle} hashes to {actual}, where the run recorded {recorded}. the bundle changed after it was protected: protect the run again with probierz protect {app} {run} <kind> --remove-source.
  • artifact-file-unnamed (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: an artifact names no file: {artifact}. record {run} again: every artifact Probierz lists names its file beside the run manifest.
  • artifact-misplaced (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: artifact is missing or escapes its run: {relative} {why}. restore {relative} below {root} from the run's bundle with probierz restore, or record {run} again.
  • artifact-hash-other (probierz.receipt.findings, exit 1, refused): checking the evidence run {run} of {app} kept: {run}: artifact hash mismatch: {relative} hashes to {actual}, where the run recorded {recorded}. the artifact changed after the run: restore it with probierz restore, or record {run} again.
  • secret-scan-failed (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: plaintext secret scan is missing or has findings: {why}. remove each credential the scan names from the run's artifacts (probierz secrets scan {root} lists them), or protect the run with probierz protect, which scans before it encrypts.
  • status-other (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: status {status}, where only a passed run is signed as evidence. fix what made {run} fail, record the journey again and sign the passing run.
  • harness-identity (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: harness source identity mismatch or incomplete: {why}. record {run} again with the harness the receipt expects, which writes harness.sha256, gitSha and worktreeSha256.
  • source-identity (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: app source identity mismatch or incomplete: {why}. record {run} again at the app source the receipt expects, from the app's checkouts.
  • build-hash (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: build hash missing or invalid: {found} is not a SHA-256. record {run} again against a build Probierz hashes (the target's build in the product's probierz.yaml).
  • artifact-hashes (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: artifact hashes incomplete or invalid: {why}. record {run} again with this Probierz, which hashes every artifact it lists.
  • level-below (probierz.receipt.findings, exit 1, refused): signing run {run} into the receipt of {app} {release}: {run}: {level} is below {minimum}, the --minimum the receipt signs. record {run} again with the evidence {minimum} asks for, or sign with the --minimum the release policy declares.
  • build-not-one (probierz.receipt.findings, exit 1, refused): signing the receipt of {app} {release}: {target}: runs do not identify one exact build: build {first} and build {second}. sign runs of one build per target: record the journeys of {target} again against the build to release.
  • journey-missing (probierz.receipt.findings, exit 1, refused): signing the receipt of {app} {release}: missing journey: {journey}, which --journeys requires, is in no passed run. record a passing run of {journey} and sign it with the others.
  • state-blank (probierz.config, exit 1, config): finding Probierz's state directory: PROBIERZ_STATE_DIR is set but blank, so it names no directory. set PROBIERZ_STATE_DIR to an absolute directory, or unset it so .probierz in the home directory is used.
  • state-relative (probierz.config, exit 1, config): finding Probierz's state directory: PROBIERZ_STATE_DIR names {path}, which is not an absolute path, so it would mean a different directory in every working directory. set PROBIERZ_STATE_DIR to an absolute directory.
  • unreadable (probierz.config, exit 1, config): reading this instance's configuration at {file}: the file cannot be read: {error}. make {file} a file this user can read, or remove it to set no key.
  • malformed (probierz.config, exit 1, config): reading this instance's configuration at {file}: the file is not a JSON object of keys: {error}. write {file} again with probierz config set, or remove it to set no key.
  • stored-unknown (probierz.config, exit 1, config): reading this instance's configuration at {file}: it holds {key}, which no part of this Probierz reads (a key a release before or after this one declared). remove it with probierz config unset {key}.
  • stored-invalid (probierz.config, exit 1, config): reading {key} from this instance's configuration at {file}: it holds {value}, which is not a {kind}. set it again with probierz config set {key} <value>, or remove it with probierz config unset {key}.
  • unwritable (probierz.config, exit 1, config): writing this instance's configuration at {file}: {step} failed: {error}. make {directory} a directory this user can write, or point PROBIERZ_STATE_DIR at one.
  • key-unknown (probierz.config, exit 2, refused): naming a configuration key: {key} is no key any part of Probierz reads. name one of the listed keys; probierz config list prints what each decides.
  • value-invalid (probierz.config, exit 2, refused): setting {key}: {value} is not a {kind}: {why}. give {key} a {kind}.
  • workspace-unset (probierz.manifest.discover, exit 1, config): finding the repositories that hold product manifests: this instance names no workspace, so there is no directory of repositories to look in. run probierz config set workspace <directory holding one checkout per repository>.
  • workspace-unreadable (probierz.manifest.discover, exit 1, config): listing {workspace}: the operating system would not list it: {error}. give this user read permission on {workspace}, or set another workspace with probierz config set workspace.
  • listing-failed (probierz.manifest.discover, exit 1, config): asking the git of {repository} which manifests it tracks: {cause}. make {repository} a git repository this user can read.
  • untracked (probierz.manifest.discover, exit 1, config): reading the manifests of {repository}: {file} is not tracked by the repository's git, so no other host and no review sees it. run git -C {repository} add {relative} and commit it, or remove the file.
  • duplicate (probierz.manifest.discover, exit 1, config): finding the manifest of {app}: two repositories declare {app}: {first} and {second}. keep the manifest of {app} in the product's own repository and remove the other.
  • not-found (probierz.manifest.discover, exit 1, config): finding the manifest of {app}: no repository under {workspace} has .probierz/{app}.yaml. add .probierz/{app}.yaml to the product's repository (probierz project adopt or probierz author manifest write it), or name an app probierz apps list lists.
  • file-name-invalid (probierz.manifest.discover, exit 1, config): finding the manifests of {repository}: {file} names app "{app}", which is not an id: letters, digits, '.', '_' and '-', starting with a letter or digit. rename it .probierz/<appId>.yaml with the id it declares, with git mv.
  • repository-name-invalid (probierz.manifest.discover, exit 1, config): finding repository "{name}" in the workspace: a manifest names a checkout directly under the workspace, and "{name}" would name another place. name the checkout by its directory name under the workspace in repositories[].repository.
  • incomplete (probierz.report, exit 69, infra_down): reading every source the report shows: {count} sections could not be read: {sections}. fix the source each section names, then run probierz report again.
  • save-failed (probierz.report, exit 1, config): keeping the report in the evidence store at {file}: {error}. make the harness's test-results directory writable.
  • send-failed (probierz.report, exit 69, infra_down): paging the report's summary through Stado's alert channels: {command} refused: {cause}. make a channel stado alerts channels lists resolve, or record the operator's channels with stado alerts preferences.
  • stado-missing (probierz.alternatives, exit 1, config): asking Stado which alternative answers: {command}: Stado could not be started: {error}. install Stado (the stado release installer at stado://releases/stado/<version>/<platform>/install.sh) so its alternatives can be asked.
  • none-answers (probierz.alternatives, exit 69, infra_down): finding an alternative of {family} for Probierz: no alternative answered {command}: {cause}. make one of the alternatives Stado names answer (stado capabilities list {family}), or select another order with stado capabilities select {family}.
  • answer-malformed (probierz.alternatives, exit 1, config): reading Stado's answer to {command}: {error}. update Stado to the release whose answer Probierz reads (stado product update stado).
  • program-failed (probierz.alternatives, exit 1, refused): {command}: {cause}. fix what the provider names, or ask Stado for the next alternative with stado capabilities resolve --failed <provider>.
  • placement-unset (probierz.alternatives, exit 1, config): opening a browser: this instance has not decided where a browser may run (browser.placement). run probierz config set browser.placement stado (only on a host Stado places the run on) or local.
  • placement-refused (probierz.alternatives, exit 1, refused): opening a browser: browser.placement is stado and this process is not a job Stado placed. run it through Stado: probierz remote run web --app <app> --host <host>, or the same verb under probierz remote.

Source snapshot#

Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.

Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.