Documentation examples as journeys.
Every `$ command` a product's documentation page shows is a journey: `probierz run examples` runs it where Stado placed the run and holds it to the lines the page says it prints.
What runs#
A product declares the page under its manifest's examples surface (surfaces.examples.spec, an https address), or names one with --spec. Probierz reads the page and takes every <pre> block as text: a line starting with $ is one example, and the lines after it, up to the next $ line, are what the page says it prints. A Markdown page with no <pre> block, as a library's README is served raw, gives its fenced code blocks instead, so a library's README examples run as its users would type them; Probierz's own Markdown page of examples is https://probierz.wisent.com/docs/examples.md, whose second example differs on purpose like the block below (real test: tests/examples/run.rs).
Each example runs with sh -c in a fresh directory of the run. It holds when every shown line is a line of what it printed, standard output and standard error together; in a shown line, <…>, … and ... stand for any text. An example that shows no line holds when it exits 0.
A run writes run-manifest.json under test-results/<app>/examples/<runId>/ like every surface, with each example's command, shown lines, printed text, exit status and the lines it did not print, so probierz history, probierz report and defects read it as any other run.
Where it runs#
An example runs the product's commands, which change state, so only a host Stado placed the run on runs them: a job Stado placed exports PROBIERZ_SOURCE_IDENTITY. Elsewhere probierz run examples is refused as not-placed and runs nothing; run it with probierz remote run examples --app <app> --host <host>.
probierz remote run examples --app <app> --host <host from probierz hosts>Refusals#
page-invalid: the spec is not an https address. Nothing runs.not-placed: the run is not in a job Stado placed. Nothing runs.page-unreadable: the page could not be fetched; the error is named and the refusal is retryable. Nothing runs.no-examples: the page shows no$ command. Nothing runs.output-differs: some examples did not print what the page shows; the refusal counts and names them, and the run is kept as evidence.
An example that differs#
The block below is this page's own example. The first example holds; the second prints a line the page does not show, on purpose, so probierz run examples --spec https://probierz.wisent.com/docs/examples is refused as output-differs naming echo probierz-example-printed and not the first. Probierz's own test of that refusal runs this page.
$ echo probierz-example-holds
probierz-example-holds
$ echo probierz-example-printed
probierz-example-shown