Docs / source reviewed
agent-evals
Combine deterministic rules and Jev judgments in agent tests
How this project uses Jev
- Input
- Test conversations, agent outcomes, tool traces and expected behavior; semantic judges use selected state
- Jev decides
- Answer structured questions such as whether a reply is grounded or satisfies a requirement, returning probabilities to semantic scorers
- Code executes
- The runner batches questions sharing a state, combines semantic and deterministic checks, produces CI verdicts and replays recorded responses
Evidence and limitations
CI replay does not revalidate the live model; recording and live judgments require service access. Missing tool traces are unknown rather than safe, and skipped checks leave the denominator, so inspect coverage. Author example thresholds and cost savings do not generalize automatically. Not independently tested here.
This project has not been run independently here. Author-reported results are not independently verified results.
Original sources
- https://github.com/marianoberton/agent-evals/blob/242ddd78864a5166b870ef43ae33ee914fd857d6/README.md ↗
- https://github.com/marianoberton/agent-evals/blob/242ddd78864a5166b870ef43ae33ee914fd857d6/src/jev/client.ts ↗
- https://github.com/marianoberton/agent-evals/blob/242ddd78864a5166b870ef43ae33ee914fd857d6/src/scorers/jevJudge.ts ↗
- https://github.com/marianoberton/agent-evals/blob/242ddd78864a5166b870ef43ae33ee914fd857d6/src/core/run.ts ↗