probierz evaluate talk

Invocation#

probierz evaluate talk --slides <SLIDES> --script <SCRIPT>

Required inputs and options#

• `--slides` is the directory of `slide-<number>.png` pictures `lem-cli talk render --slides` writes, numbered from the first without a gap; `--script` is the speaker script Markdown (`.md`) `lem-cli talk render --notes` writes; `--source` names a document the talk must stay true to (Markdown, text, TeX or PDF) and is repeated for each — at least one is required. A PDF source is read through `pdftotext` (poppler).
• `--rubric` names a rubric file (`name` and non-empty `dimensions`, each with a `criterion`); without it the built-in rubric for a talk given to a review panel is used. There are no weights, minimums or overall score.
• The vision model comes from `--model` or `PROBIERZ_TALK_MODEL`; the router from `--router-url` or `STADO_MODEL_ROUTER_URL`, its bearer from `STADO_MODEL_ROUTER_TOKEN` or `--router-token-stdin`, and the agent identity from `--agent-id` or `PROBIERZ_MODEL_AGENT_ID` with `PROBIERZ_MODEL_AGENT_SECRET`, both or neither.

Output and state effects#

Shows the model every slide picture in order, the speaker script and the full text of every source, and asks it to judge each rubric criterion met or not with its evidence. The talk passes only when every criterion is met and the model names no blocker; an unmet criterion is reported as the blocker `criterion_unmet:<name>` with its evidence. Writes the report once to `--out`, or to `test-results/talk-evaluations/<timestamp>-<script>.probierz.json`, naming what was judged (slides, script, sources and rubric). MCP: `probierz_evaluate_talk`.

Exact refusals#

• `--source <document> is required: the talk's claims are judged against the documents it presents`, and `--script must be the speaker script Markdown (.md): <path>`.
• `--slides <dir> holds no slide-<number>.png pictures; render them with lem-cli talk render --slides`, two pictures of one slide (`… two pictures of slide <n>: keep one`), a gap (`--slides <dir> has no slide-<n>.png: the slides must run from the first without a gap`), and a picture whose name gives no place in the talk.
• `--source <path> must be Markdown, text, TeX or PDF`, `--source <path> holds no text to judge the talk against`, and a `pdftotext` that cannot start or read the PDF, named with its error.
• `--model or PROBIERZ_TALK_MODEL names the vision model that judges the talk`, `STADO_MODEL_ROUTER_TOKEN or --router-token-stdin gives the router bearer`, and `agent identity needs both an agent ID and an agent secret`; a router that refuses or answers without the one tool call is named with its status and sentence (exit 75, unavailable).
• `talk rubric needs a non-empty name`, `talk rubric dimensions must be a non-empty object`, `talk rubric dimension <name> needs a non-empty criterion`, and `the talk evaluation does not judge <name>.met` when the model leaves a criterion unjudged.
• `--out must end in .json`, and `the talk evaluation report already exists: <path>`: a report is never overwritten.

Source snapshot#

Generated from the Rust binaries at https://github.com/wisent-ai/probierz.git revision `f65b755da397f2ec7435eb80ed722aa6017437c1`.

Combined help SHA-256: `4a12655d233812abc29eeebbf3db3065efe3e00288024383f3eef3970d092e94`; binary SHA-256: `d5398832d3ab52c541b1efe979730dea62941ac6197898a20fccfc5d3fca44df`.