Docs / source reviewed
jev-eval
Measure Jev confidence and escalation thresholds on your own labels
How this project uses Jev
- Input
- Labeled JSONL states, gold answers and Choice, Noul or Score question definitions; model calls send only states and questions
- Jev decides
- Jev returns structured judgments per row, then code compares them with labels to calculate accuracy, calibration error and the coverage/error tradeoff at different gates
- Code executes
- Stores resumable result caches and compares models or question wordings. Holdout or bootstrap out-of-bag evaluation supports threshold recommendations; fewer than 50 labels causes refusal and fewer than 200 marks results provisional
Evidence and limitations
The MIT license, HTTP integration, analysis and threshold-export logic were reviewed without live model evaluation. Recommendations are for human review and never change production gates automatically. Label quality, sample size and distribution shifts affect validity. Calibration statistics neither guarantee future accuracy nor improve the model itself. README examples and general accuracy claims are not Gallery measurements.
This project has not been run independently here. Author-reported results are not independently verified results.
Original sources
- https://github.com/CMaintz/jev-eval/blob/11188137813bceab2df2e173da15f9d0872ab995/README.md ↗
- https://github.com/CMaintz/jev-eval/blob/11188137813bceab2df2e173da15f9d0872ab995/LICENSE ↗
- https://github.com/CMaintz/jev-eval/blob/11188137813bceab2df2e173da15f9d0872ab995/src/jev_eval/provider.py ↗
- https://github.com/CMaintz/jev-eval/blob/11188137813bceab2df2e173da15f9d0872ab995/src/jev_eval/analysis.py ↗
- https://github.com/CMaintz/jev-eval/blob/11188137813bceab2df2e173da15f9d0872ab995/src/jev_eval/thresholds.py ↗
- https://github.com/CMaintz/jev-eval/blob/11188137813bceab2df2e173da15f9d0872ab995/src/jev_eval/contract.py ↗