Author test

Convex Coding Evals — Decision Model Evaluations

Evaluate Convex coding knowledge with Jev decision questions

get-convex · Sources checked 2026-10-09

Added to this collection 2026-10-09

How this project uses Jev

Input: A human-authored answer bank for Convex backend coding tasks, four-option questions and optional context conditions
Jev decides: Jev selects one of four options; the evaluator validates responses, grades against the reference answer and uses matched option permutations
Code executes: Decision-model scores are reported separately from code-generation evaluations; the question bank contains 108 questions spanning 90 coding-eval sources, weighted by source evaluation

Evidence and limitations

The Apache-2.0 license, question-bank manifest, Jev’s OpenRouter integration and evaluation flow were reviewed without rerunning the benchmark. Multiple-choice scores measure specific Convex knowledge, not the ability to write or debug a working application. The README says five questions have clean-checkout replay commands while 103 still need portability work; the Gallery did not reproduce the authors’ results.

This project has not been run independently here. Author-reported results are not independently verified results.

Original sources

Explore related use cases

Browse this category →