WisentProbierz
Probierz/Docs

CLI command

probierz evaluate security

Execute declared security checks only against an owned application, optionally asking an explicit Brama model to prioritize them and explain results.

Invocation#

Shell
probierz evaluate security --app <APP_ID> --out <OUTPUT>

Required inputs and options#

  • `--app` and `--out` are required. Choose exactly one of `--model <route>` and `--deterministic`. An unrestricted or abliterated model uses the same authenticated Brama gateway as other models. Deterministic mode executes checks in declaration order without contacting Brama or reading model credentials; it is an explicit choice, never a fallback after model failure.
  • The registered application's `apps/<id>/probierz.yaml` must declare `security` with `owner` (exactly matching the manifest owner), nonempty `authorization`, credential-free `origin`, exact approved IP `addresses`, positive `response_bytes`, and a nonempty list of unique `checks`. Only an operator-maintained manifest authorizes scope; a model cannot change it.
  • A baseline check is `{kind: headers, id: public, path: /}` or `{kind: access, id: private, path: /private, expected_status: [401, 403]}`. Paths must be canonical absolute paths, without query, fragment, percent escapes, traversal or another authority. `access` permits only declared 401, 403 or 404 denial responses; choose the exact denial contract of your application.
  • `policy` checks also declare `expected_status` and nonempty `headers` assertions: `{expect: present, name: content-security-policy}`, `{expect: absent, name: server}`, or `{expect: equals, name: x-content-type-options, value: nosniff}`. Equality is byte-exact and requires one header value; repeated or conflicting headers cannot satisfy it. Header names are case-insensitive and may appear only once per policy. Invalid names, injected line breaks and empty equality values are refused.
  • Failed policy assertions retain `header`, a machine-readable `reason`, and a human-readable `cause`. Reasons distinguish `missing`, `unexpected`, `repeated`, and `value-mismatch`. Repeated exact-value headers are reported as repeated even when one value matches. Response header values are never included in these diagnostics.
  • `cors` checks declare `expected_status` and `denied_origin`, for example `{kind: cors, id: foreign-origin, path: /private, expected_status: [403], denied_origin: https://foreign.invalid}`. This sends a single GET with that Origin header only to the owned target. The denied origin is never resolved or contacted. It must be a canonical HTTP(S) origin different from the owned origin; paths, credentials, queries and fragments are refused.
  • `cookies` checks declare `expected_status` and a nonempty list of `cookies`, each with `name`, `secure`, `http_only` and `same_site` (`strict`, `lax`, `none` or `unchecked`). Every field is required; `unchecked` states explicitly that SameSite is not asserted. Example: `{kind: cookies, id: session-cookie, path: /login, expected_status: [200], cookies: [{name: session, secure: true, http_only: true, same_site: lax}]}`. Names must be RFC 6265 tokens and unique per check; `same_site: none` requires `secure: true`. The probe keeps no cookie jar and never sends cookies.
  • Each cookie observation reports `failedCookies` entries with `cookie`, `reason` and `cause`. Reasons are `missing`, `repeated` (the response sets the declared name more than once, which is ambiguous and refused rather than guessed), `not-secure`, `not-httponly` and `samesite-mismatch`. Cookie values and attribute values are never retained. A denied-origin CORS finding carries the reason `denied-origin-allowed`.
  • `STADO_MODEL_ROUTER_URL` and `STADO_MODEL_ROUTER_TOKEN` configure Brama; `PROBIERZ_MODEL_AGENT_ID` and `PROBIERZ_MODEL_AGENT_SECRET` supply the signed workload identity. URL and agent flags override their environment values. `--router-token-stdin` reads the bearer and signing secret as consecutive lines. Provider credentials remain in Brama, not Probierz.
  • Example without model inference: `probierz evaluate security --app owned-service --deterministic --out test-results/security-deterministic.json`. This still performs real GETs against the explicitly owned target. Its report records `mode: deterministic`, with null `modelRequested`, `plan` and `analysis`; it does not claim a model reviewed the findings.
  • Example: `probierz evaluate security --app owned-service --model abliteration-ai/abliterated-model --out test-results/security-assessment.json`. The provider route must be entitled and available; catalogue presence alone is not proof of inference.

Output and state effects#

Performs one GET per declared check. DNS is resolved and every address checked against the declaration, then pinned for the run; proxies and redirects are disabled. A headers check requires a successful status and the presence of CSP and X-Content-Type-Options, plus HSTS on HTTPS. Access checks compare the actual unauthenticated status to the declared denials. Policy checks require the declared status and all header assertions. CORS checks require the declared status without an Access-Control-Allow-Origin header naming the denied origin or a wildcard; allowing a different explicit origin does not grant the denied origin. Bodies and response header values are discarded. Only check IDs and paths enter model planning, never declared header equality values. The model orders all checks exactly once and supplies non-executable advice. The immutable report retains exact source identity, approved scope, observations, model request/response hashes, usage, completion and the actual error; failedAssertions names failed headers and causes. Exit is unsuccessful on a finding or incomplete run; complete distinguishes execution failure from completed findings. These checks do not establish whole-system security, parse the semantics of CSP directives, test CORS preflight or authorize exploitation.

Exact refusals#

  • Omitting both modes, selecting both, or combining `--deterministic` with router URL, agent ID or token-stdin flags is a usage refusal before any run report is created. Inherited model configuration is ignored in deterministic mode.
  • Missing ownership or authorization, mismatched manifest owner, invalid origins, empty address lists, duplicate IDs and escaping paths are refused before any run. Unknown declaration fields are errors.
  • Any DNS address outside the approved set stops the run before target or model requests. A redirect is observed but never followed. Responses exceeding the operator-declared byte bound fail rather than being graded after truncation.
  • In model-assisted mode, missing Brama credentials, unavailable routes, HTTP refusals, malformed model JSON, omitted, duplicate or invented check IDs and advice referring to undeclared checks retain an incomplete report; they never become a pass or silently switch to deterministic mode.
  • An existing report path is refused and left unchanged. Parent directories must already exist. A malformed declaration is reported on stderr without creating a run report.
  • Each run also creates `<report-path>.events.jsonl` without overwriting an existing journal. Initialization, operation starts and completed observations are appended and synced before later network work. If the process is interrupted, the final JSON report may be empty, but complete journal lines remain readable; an unterminated final line is partial evidence, not a completed event. The journal records progress, not a passing assessment.
  • An incomplete report names `failedOperation` and, when applicable, `failedCheck`. Operations distinguish scope/DNS resolution, model configuration, model planning, target checks, model advice and journal creation. Journal write failures stop further checks; the error is retained in the final report when that destination remains writable. If an existing journal prevents startup, it stays unchanged and the new report explains the refusal.
  • The deterministic live journey checks that the journal retains every observed target outcome, leaves existing reports and journals unchanged, and records model-configuration refusal before any target check. Native qualification reads the actual journal, refuses events after a terminal record, and treats an unterminated final record as partial rather than successful execution.
  • Desktop: Evidence → Security runs this same command, displays its report and diagnostics, and can open retained security reports. MCP: `probierz_evaluate_security` uses the same evaluator and environment credentials.
  • The real security journey additionally requires `PROBIERZ_SECURITY_TEST_EXPECTATIONS`, a JSON file mapping each declared check ID to its independently declared `{pass, status}` outcome. The isolated real target must cover `policy`, `cors` and `cookies` checks and both a passing check and a security finding. The journey compares every observed verdict and HTTP status to that target contract, not to expectations inferred from the tested report.
  • Each policy expectation also declares `failedAssertions`, an ordered array of `{header, reason}` entries, and each cookie expectation declares `failedCookies`, an ordered array of `{cookie, reason}` entries; `[]` means every assertion passes. Both live modes and the native journey compare those classifications to the independent target contract without pinning human-readable wording, and refuse any finding that carries a response value. Qualification targets should exercise absent required headers, present forbidden headers, repeated equality headers, unequal single values, a missing cookie and a cookie lacking a declared attribute.
  • Real journey: `cargo test --manifest-path probierz-rs/Cargo.toml --features security-live-tests --test security_journey`, with `PROBIERZ_SECURITY_TEST_HARNESS`, `PROBIERZ_SECURITY_TEST_APP` and `PROBIERZ_SECURITY_TEST_MODEL` naming the real harness, authorized application and available Brama model. The normal Brama credential variables are required. It retains commands, exit statuses, reports and diagnostics beneath `.build/security-results/`, verifies every check ran once, refuses report overwrite, and refuses a separate test-owned manifest with mismatched ownership. No provider or target is simulated.
  • To qualify deterministic mode independently of Brama, use the same command with the test filter `real_deterministic_assessment_requires_no_model_credentials`. This needs the real harness, app and independently declared expectations, but no model or model credentials. It strips model environment settings from the CLI process, verifies every actual status and verdict in declaration order, and covers immutable evidence and conflicting-mode refusals.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.