CLI command

probierz oracle calibrate

Measure a model oracle on cases with a known verdict.

Invocation#

Shell
probierz oracle calibrate <ORACLE> --app <APP> --cases <FILE>

Required inputs and options#

  • <ORACLE> is screen or rubric; --app names whose model settings, and for rubric whose rubrics, it uses; --cases is a JSON list of cases, each with the verdict a person gave (pass|fail): {screen, promise, verdict} for screen, {rubric, question, answer, verdict} for rubric, the rubric a file beside the app's manifest.

Output and state effects#

Judges every case as judge would (a rubric answer passes at its rubric's threshold) and keeps the share it got right with the cases' digest and the models that answered; once it reaches oracle.accuracy-minimum the oracle decides alone.

Exact refusals#

  • cases-invalid (exit 1): an oracle other than screen or rubric, a cases file that is not a list, one with no case, or cases without the verdict a person gave or the input the oracle is asked (a rubric that does not resolve included) are refused naming every such case at once. Every case is read before the first model call, so nothing is asked and no calibration is recorded. Real test: tests/oracle/calibrate.rs.
  • A rubric with no numeric threshold, or a grade without a numeric score, is refused as for judge.

Source snapshot#

Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.

Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.