CLI command
probierz oracle calibrate
Measure a model oracle on cases with a known verdict.
Invocation#
probierz oracle calibrate <ORACLE> --app <APP> --cases <FILE>Required inputs and options#
<ORACLE>isscreenorrubric;--appnames whose model settings, and forrubricwhose rubrics, it uses;--casesis a JSON list of cases, each with the verdict a person gave (pass|fail):{screen, promise, verdict}forscreen,{rubric, question, answer, verdict}forrubric, the rubric a file beside the app's manifest.
Output and state effects#
Judges every case as judge would (a rubric answer passes at its rubric's threshold) and keeps the share it got right with the cases' digest and the models that answered; once it reaches oracle.accuracy-minimum the oracle decides alone.
Exact refusals#
cases-invalid(exit 1): an oracle other thanscreenorrubric, a cases file that is not a list, one with no case, or cases without the verdict a person gave or the input the oracle is asked (a rubric that does not resolve included) are refused naming every such case at once. Every case is read before the first model call, so nothing is asked and no calibration is recorded. Real test: tests/oracle/calibrate.rs.- A rubric with no numeric
threshold, or a grade without a numeric score, is refused as forjudge.
Source snapshot#
Generated from the Rust binaries at revision 99a74945a3598a8d73b6c6cbe5aae4f283becea3 of Probierz's private source repository.
Combined help SHA-256: 77529678a61237f774b59e76df0d1907cb749ef5da56d2ed913b040793b7d30b; binary SHA-256: be9e0eaff0e6ea50403a0c6f6d467697a4eac676f9523b8dc13a596d2b5526c5.