CLI command
probierz oracle judge
Judge one recorded run with every oracle and keep the verdicts.
Invocation#
probierz oracle judge <APP_ID> <RUN_ID>Required inputs and options#
<APP_ID>is the product, asprobierz apps listnames it.<RUN_ID>is a runprobierz historylists.
Output and state effects#
console and network read events.json; layout, links and leaks read what a web journey program kept on each live screen as screens/<name>.json (the value of the one expression probierz author spec gives every web program: the page's address, viewport and device pixel ratio, its layout problems and its links), so a screen is judged with its own styles, viewport and fonts. Layout fails on text a box clips, a page wider than its viewport, and controls drawn over one another (a control inside another is one control); one pixel is rounding, since CSSOM View rounds both widths. Links asks every link to the page's own origin once, HEAD and GET where HEAD is not answered (405 or 501), with the run's traffic marker, and fails on an answer of 400 or above or none; links to other origins are counted, never followed. Leaks fails on a JSON web token, private key, AWS access key, GitHub or Slack token (read as secrets scan reads them), bearer credential, credential or e-mail address in a query, or password in an address, found anywhere in events.json, the screens' facts and element trees, or the app logs the native harness kept (logs/<name>.log); each is named by kind, file, place (a JSON pointer or a line) and length, never shown. A native screen (iOS, Android, macOS, Windows) is judged by the element tree the harness kept as screens/<name>.xml: layout fails on controls drawn over one another; accessibility, at every run, on a control a screen reader cannot name (by its own label or one inside it, as TalkBack and VoiceOver read a container) and on a target smaller than its platform asks (48x48 dp on Android, from its display density; 44x44 pt on iOS, Apple's Human Interface Guidelines; 24x24 px on macOS, WCAG 2.2 2.5.8), Windows' physical pixels leaving target size named as unchecked; links is none, since a native screen has no links. Which elements are controls is read from the platform the screen's facts name (android, ios, mac or windows, as Appium reports it); a screen whose facts name another platform or none is named as unchecked and judged by neither layout nor accessibility, and when no screen could be judged both are none with that reason, never a pass of nothing. On the web, accessibility audits each screens/<name>.html (kept with a <base> of the page's address, so its styles load) with axe-core through the browser alternative's weles_page_accessibility, running the WCAG rules of the journey's declared level (A: axe's wcag2a and wcag21a; AA adds wcag2aa, wcag21aa and wcag22aa; AAA adds wcag2aaa) and keeping each violation with its rule, impact, WCAG tags and the elements it names, beside the axe version. Baseline compares each screens/<name>.png pixel by pixel with the one accepted in the run's matrix cell (baseline accept), and the share that differs at all against the manifest's baselineTolerance (none when it declares none), naming the cell and, for a screen with no baseline there, the baseline accept command that sets one; the regions the manifest's baselineMask entries cover are neither compared nor counted: a CSS selector on a web screen, found in the kept document laid out at the screen's live viewport and scaled by its device pixel ratio, and ~<id> on a native one, each shown element whose identifier or label is <id>, scaled by the image's width over the window's. A screen whose mask cannot be placed (no document, tree or facts kept) is not judged. Screen asks a model whether the screen shows what the journey promises; the same answer lists every misspelling, grammar error or wrong word of the screen's text, judged in its own language, as the language verdict (each finding with its screen, quote, problem and model), which is always advisory-pass or advisory-fail since no calibration measures it, and none when screen is none. Rubric grades answers.json on the journey's rubric. Each verdict is pass, fail, or none with the reason its input is missing; a model oracle below oracle.accuracy-minimum gives advisory-pass or advisory-fail. Product text reaches the model fenced as untrusted, and each verdict names the model that gave it. A browser alternative without the audit refuses it, and the run's oracles are kept as unjudged with that sentence. Real test of the language verdict: tests/oracle/language.rs, through the host's Stado.
Exact refusals#
run-unknown(probierz.records.run, exit 1, the one refusal of a run nobody recorded wherever a command names a run): a run of that id not recorded for the product is refused namingprobierz history; nothing is judged. Run evidence that is there but cannot be read is refused naming the file, never taken for an absent run. Real test: tests/oracle/judge.rs.evidence-unreadable(exit 1): a screen, element tree, log, facts file or accepted baseline the run is judged by that cannot be read or listed is refused naming the path and the operating system's answer; the run is not judged without it and no verdict is kept.rubric-invalid(probierz.oracle, exit 1): the rubric the journey names is not under.probierz/<app>/, or declares no numericthreshold(0 to 1), without which every answer would pass. It is read before any oracle runs, so no browser is opened and no model asked; incalibrate rubricit is named among thecases-invalidproblems. Real test: tests/oracle/judge.rs.- A model grade that carries no numeric score, or a screen verdict that is neither pass nor fail, is refused naming what the model gave, never counted as zero or as a pass.
- A
baselineTolerancethat is not a number from 0 to 1, or abaselineMaskthat is not a list of CSS selectors and~<id>entries, ismanifest-invalidwhen the manifest is loaded, before anything is judged (seeapp show); a selector the browser cannot parse is refused with the browser's sentence.
Source snapshot#
Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.
Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.