CLI command

probierz bisect

The first change that broke a journey, found by running it on the revisions between its last green run and its first red one; or a flaky journey quarantined until a date.

Invocation#

Shell
probierz bisect <APP_ID> <JOURNEY> --host <HOST> [--good <REVISION>] [--bad <REVISION>]
probierz bisect <APP_ID> <JOURNEY> --quarantine-until <YYYY-MM-DD>

Required inputs and options#

  • <APP_ID> and <JOURNEY> are required; --host is the Stado host every step runs on.
  • --good and --bad default to the revisions of the journey's newest green and red runs recorded at a commit: a suite run's source.revision, or a surface run's primary repository gitSha, when its source carried no changes not committed (a run of uncommitted changes is at no commit and is passed over). --quarantine-until instead keeps the journey running and reported but not blocking until that date.

Output and state effects#

Runs both ends again first; when both reproduce, halves the commits between them, each step a probierz remote run --revision <commit> that keeps no defect, and names the first commit whose run failed. A step counts only when its run happened: a job that completed passed, one that failed with the journey's evidence on a ready host failed. The result is kept and shown by probierz report beside the journey.

Exact refusals#

  • no-green (exit 1): no passing run of the journey recorded at a commit and no --good; no-red (exit 1): no failing run recorded at a commit and no --bad. Each is refused naming the fix before anything runs. Real test: tests/bisect/run.rs.
  • Ends that do not reproduce blame no commit: both runs are kept, and the class probierz report gives the failure (data, infrastructure, dependency) is where to look.
  • step-unrun (exit 69, worth retrying): a step whose run did not happen (not submitted, no evidence collected, a host not ready) is refused naming the revision, the state and the cause; no commit is blamed for it.
  • quarantine-invalid (exit 2): --quarantine-until that is not a date as YYYY-MM-DD, or not after today; nothing is quarantined and the journey keeps blocking.

Declared refusals#

What probierz refusals list declares for this command at revision d2cd6156e8b8dc6204dab7b46ec0b34b2bb03cea (probierz 0.1.7 (revision unknown, not built from a git checkout)): each cause's id, the operation declaring it, its exit status and code, and its remedy.

  • workspace-unset (probierz.manifest.discover, exit 1, config): finding the repositories that hold product manifests: this instance names no workspace, so there is no directory of repositories to look in. run probierz config set workspace <directory holding one checkout per repository>.
  • workspace-unreadable (probierz.manifest.discover, exit 1, config): listing {workspace}: the operating system would not list it: {error}. give this user read permission on {workspace}, or set another workspace with probierz config set workspace.
  • listing-failed (probierz.manifest.discover, exit 1, config): asking the git of {repository} which manifests it tracks: {cause}. make {repository} a git repository this user can read.
  • untracked (probierz.manifest.discover, exit 1, config): reading the manifests of {repository}: {file} is not tracked by the repository's git, so no other host and no review sees it. run git -C {repository} add {relative} and commit it, or remove the file.
  • duplicate (probierz.manifest.discover, exit 1, config): finding the manifest of {app}: two repositories declare {app}: {first} and {second}. keep the manifest of {app} in the product's own repository and remove the other.
  • not-found (probierz.manifest.discover, exit 1, config): finding the manifest of {app}: no repository under {workspace} has .probierz/{app}.yaml. add .probierz/{app}.yaml to the product's repository (probierz project adopt or probierz author manifest write it), or name an app probierz apps list lists.
  • file-name-invalid (probierz.manifest.discover, exit 1, config): finding the manifests of {repository}: {file} names app "{app}", which is not an id: letters, digits, '.', '_' and '-', starting with a letter or digit. rename it .probierz/<appId>.yaml with the id it declares, with git mv.
  • repository-name-invalid (probierz.manifest.discover, exit 1, config): finding repository "{name}" in the workspace: a manifest names a checkout directly under the workspace, and "{name}" would name another place. name the checkout by its directory name under the workspace in repositories[].repository.
  • app-id-invalid (probierz.manifest.load, exit 2, refused): naming an app: "{app}" is not an id: letters, digits, '.', '_' and '-', starting with a letter or digit. name an app probierz apps list lists.
  • manifest-unreadable (probierz.manifest.load, exit 1, config): reading {file}: {error}. write {file} as YAML the reader accepts (its line and column are named), then load it again.
  • manifest-invalid (probierz.manifest.load, exit 1, config): judging {file}: {problem}. fix the declaration in {file}; probierz app check names every problem it has with the world once it loads.
  • manifest-id-mismatch (probierz.manifest.load, exit 1, config): loading the manifest of {app}: {file} is named for {app} and declares appId {declared}. declare appId: {app} in {file}, or rename the file for the app it declares with git mv.
  • name-invalid (probierz.schedule, exit 2, refused): naming a schedule: {name} is not a name: lower-case letters and digits, in words joined by single dashes. name the schedule like wisent-app-nightly.
  • flag-inapplicable (probierz.schedule, exit 2, refused): adding schedule {name}: {flag} means nothing to {what}, which would fire without it. {remedy}.
  • host-unpinned (probierz.schedule, exit 2, refused): adding schedule {name} on {host}: a schedule fires on one registry host, and {host} names none. give --host stado:<target> with a target probierz hosts lists.
  • exists (probierz.schedule, exit 1, refused): adding schedule {name}: Stado already holds a schedule of that name as {id}. change it with probierz schedule edit {name}, or remove it first with probierz schedule remove {name}.
  • unknown (probierz.schedule, exit 1, not_found): finding schedule {name}: no schedule Stado holds carries that name. name one probierz schedule list shows.
  • trigger-unavailable (probierz.schedule, exit 1, refused): adding schedule {name} on {trigger}: Stado publishes no {trigger} event a schedule can wait for. schedule on a cadence (--cron) or on main (--trigger main) until Stado publishes {trigger} events.
  • journey-unserved (probierz.schedule, exit 1, config): scheduling journey {journey} of {app}: no surface of the manifest serves {journey}{target}. name a journey and target the manifest declares, or declare it under surfaces.<target>.journeys.
  • stado-refused (probierz.schedule, exit 1, refused): {command}: {cause}. fix what Stado names, then repeat the command.
  • state-unwritable (probierz.schedule, exit 1, config): recording the revisions schedule {name} has seen in {file}: {error}. make Probierz's state directory writable, or point PROBIERZ_STATE_DIR at one.
  • state-invalid (probierz.schedule, exit 1, config): reading the revisions schedule {name} has seen in {file}: the file holds no map of repositories to the revisions it saw: {error}. remove {file} to start the schedule over; its next tick records the revisions and runs nothing.
  • main-untracked (probierz.schedule, exit 1, config): reading main of {repository} for schedule {name}: main tracks no remote branch ({cause}), so no revision of the remote's main can be fetched. set it with git -C {repository} branch --set-upstream-to <remote>/main main.
  • fetch-failed (probierz.schedule, exit 69, infra_down): fetching {upstream} into {repository} for schedule {name}: {cause}. make the remote main tracks reachable from this host, then let the schedule tick again.
  • diff-failed (probierz.schedule, exit 1, config): listing what changed in {repository} between {old} and {now}: {cause}. fetch the history that holds {old} into {repository}, or remove the schedule's state to start it over.
  • runs-failed (probierz.schedule, exit 1, refused): running what schedule {name} fires: {count} of its runs did not pass: {runs}. read each run with probierz history <app>; a product failure is a defect, a test failure a repair.
  • app-required (probierz.schedule, exit 2, refused): adding schedule {name} on {trigger}: a {trigger} schedule runs one product's journeys, and no --app names it. name the product with --app; only --report and --trigger gaps cover every product.
  • gap-unfillable (probierz.schedule, exit 1, config): authoring what schedule {name} found missing: {count} gaps cannot be authored without what they lack: {gaps}. declare what each gap names (a catalog description, a test environment's baseUrl, the surface's TUI_CMD or CUA_APP_EXECUTABLE condition, the surface an escaped error was met on; an error's source must give its surface and message), or author it by hand with the command it names.
  • link-required (probierz.schedule, exit 2, refused): adding schedule {name} on store: a store schedule runs the product's journeys when the store sells a new version, and no --link names the app's store page. name the app's App Store page with --link https://apps.apple.com/<country>/app/<name>/id<number>.
  • release-unreadable (probierz.schedule, exit 2, refused): watching the releases of {package} for schedule {name}: Google publishes no version of an app a program may read, so a new release of {package} on Google Play cannot be seen. watch the app's App Store link if it is sold there too, or run its Android journeys on a cadence schedule.
  • surface-missing (probierz.schedule, exit 1, config): finding what a new release of {app} runs for schedule {name}: {app} declares no journey on a {surface} surface, so a release would run nothing. declare the journeys a phone runs under surfaces.{surface} in the product's manifest.
  • no-green (probierz.bisect, exit 1, not_found): bisecting {journey} of {app}: no passing run of it is recorded at a commit (a run of changes not committed is at none), so there is no good revision to start from. name a good revision with --good, or run the journey green once on an older revision with its changes committed.
  • no-red (probierz.bisect, exit 1, not_found): bisecting {journey} of {app}: no failing run of it is recorded at a commit (a run of changes not committed is at none), so there is no bad revision to end at. name the revision it fails on with --bad, or commit the changes it fails on and run it there so its red run is recorded.
  • ends-disagree (probierz.bisect, exit 1, refused): bisecting {journey} of {app} between {good} and {bad}: the good end did not pass or the bad end did not fail when run again: {observed}. the failure is not in the source between them: read the class probierz report {app} gives it (data, infrastructure, dependency).
  • step-unrun (probierz.bisect, exit 69, infra_down): running {journey} of {app} at {revision}: the run did not happen: it ended {state} without evidence of the journey ({failure}). fix what the run names (the host, Stado, the checkout of that revision), then bisect again.
  • quarantine-invalid (probierz.bisect, exit 2, refused): quarantining {journey} of {app} until {until}: {problem}. give --quarantine-until a date after today as YYYY-MM-DD.
  • link-invalid (probierz.store, exit 2, refused): reading the store link {link}: it is no https link to an app's page on the App Store (an id<number> under apps.apple.com) or on Google Play (play.google.com/store/apps/details with id=<package>). give the app's page as the store shares it: https://apps.apple.com/<country>/app/<name>/id<number> or https://play.google.com/store/apps/details?id=<package>.
  • app-unknown (probierz.store, exit 1, not_found): finding the app {link} names: the App Store's {country} storefront holds no app id{id}: it was removed, or it is sold only in other countries. open the link in a browser to see whether the app is still sold; an app sold elsewhere is found through a link carrying that country's code, like https://apps.apple.com/pl/app/<name>/id{id}.
  • lookup-unanswered (probierz.store, exit 69, infra_down): asking Apple's lookup {address} for the app: {problem}. check that this machine reaches itunes.apple.com over HTTPS (a proxy in HTTPS_PROXY is used when set), then ask again; an answer that is not Apple's record is said as it came.
  • page-unreachable (probierz.store, exit 69, infra_down): reading the store page {page}: it could not be reached or read: {error}. check that this machine reaches the store over HTTPS (a proxy in HTTPS_PROXY is used when set), then ask again.
  • page-unread (probierz.store, exit 1, not_found): reading the links of the store page {page}: {problem}. open the page in a browser: an app the store no longer sells answers an error, and a page whose links are there but unread is a change of the store's page that this command must follow.
  • reviews-unpublished (probierz.store, exit 1, refused): reading the reviews of {package} on Google Play: Google publishes no reviews feed a program may read; its reviews are read only through the Google Play Developer API with the developer's own account. read them in the Play Console of the app's developer, or give the app's App Store link when it is sold there too.
  • reviews-unanswered (probierz.store, exit 69, infra_down): reading Apple's reviews feed {address}: {problem}. check that this machine reaches itunes.apple.com over HTTPS (a proxy in HTTPS_PROXY is used when set), then ask again; an answer that is not the feed is said as it came.
  • store-undeclared (probierz.store, exit 1, config): reading where {app} is sold: its manifest names no App Store page (inventory.appStore). write the app's page into the manifest: inventory: {appStore: https://apps.apple.com/<country>/app/<name>/id<number>}.
  • release-unread (probierz.store, exit 1, config): reading the version {app} released from {file}: {problem}. give the product's checkout a .wisent-release.json whose version_source (kind regex, a path and a pattern with a version group) finds the version in that file, as the release pipeline reads it.
  • versions-differ (probierz.store, exit 1, refused): holding {app}'s App Store page to the version it released: the store sells {store} and the release file states {released}: the store is {verdict}. behind: submit the released version, or see why App Review has not approved it; ahead: bring the checkout to the revision that was released.
  • description-unread (probierz.store, exit 1, config): reading the App Store description {app} keeps in {file}: {problem}. keep the description the App Store page should show in a file of the product's repository and name its path in the manifest: inventory: {appStoreDescription: <path>}.
  • listing-differs (probierz.store, exit 1, refused): holding {app}'s App Store page to the description it keeps: the store shows another description than {file} holds. submit the description in the file with the next version, or bring the file to what the store shows if the store is right.
  • notes-unread (probierz.store, exit 1, config): reading the App Store release notes {app} keeps in {file}: {problem}. keep the notes of the released version in a file of the product's repository and name its path in the manifest: inventory: {appStoreReleaseNotes: <path>}.
  • notes-differ (probierz.store, exit 1, refused): holding {app}'s App Store page to the release notes it keeps: the store shows other notes for the version on sale than {file} holds. submit the notes in the file with the version, or bring the file to the notes the store shows if the store is right.

Source snapshot#

Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.

Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.