semantic-assert
Check semantic requirements against captured page or response state
Implementation and evidence →JEV GUIDE
Jev + Playwright semantic testing: assertions about page state
A message can be correct without matching one fixed sentence. semantic-assert evaluates a claim about captured state, while your test code owns acceptance. This guide combines a source review with a synthetic test design; it is not a live benchmark.
Which assertions benefit from a semantic judge?
Meaning-based checks can assess whether an error explains a cause or offers a next step. Exact totals, counts, status codes and permission boundaries still need deterministic assertions. A payment confirmation on screen does not prove that the payment service recorded a transaction.
| Question | Suggested check | Evidence |
|---|---|---|
| Does the total equal its components? | Deterministic arithmetic | Business data |
| Does the error explain the next step? | Supplementary semantic assertion | Error text and state |
| Was the form actually saved? | API assertion plus UI feedback | Persisted data and captured page |
How captured state becomes a result
The core Judge accepts JSON state and claims. A TypeSafe provider supplies model judgments; the Judge applies thresholds in code. The Playwright adapter supplies page capture and reporting integration.
When a claim fails and time remains, the engine captures state again. timeoutMs: 0 evaluates a static response once. Give changing pages a bounded polling policy instead of repeatedly asking until a favorable answer appears.
Source: semantic-assert — judge.ts ↗
Build a small acceptance set
These are synthetic editorial fixtures, not observed Jev outputs. Include different wordings of a valid message and superficially positive messages with contradictory backend evidence.
| Synthetic state | Human expectation | Purpose |
|---|---|---|
| “Upload failed: over 10 MB. Compress and retry.” | Explains cause and action | Clear positive |
| “Something went wrong.” | No actionable next step | Vague negative |
| “Saved” on screen, HTTP 500 from save API | Business failure | UI meaning cannot override backend failure |
| Empty error region while loading | Wait or capture failure | Do not judge missing state as a normal result |
Integrate without losing failure evidence
First assert that the relevant region exists, capture only the needed state, and redact it before calling a provider. Give each claim one clear condition.
- Use FakeProvider to exercise the harness. Scripted responses do not evaluate meaning.
- Record the model, claim version, state, raw value, threshold and poll count.
- Retry only when the state can change, with an explicit call and time budget.
- Preserve failure attachments rather than retaining only a binary result.
Source: semantic-assert — README.md ↗
Choose a threshold with labeled examples
Compare false accepts and false rejects on positive, negative and borderline fixtures while holding the model and question version fixed. A threshold expresses your policy, not guaranteed accuracy. Start with advisory results; retain deterministic gates for money, data integrity and authorization.
If identical state produces unstable verdicts, investigate ambiguous claims and missing context before changing the threshold. Selecting a favorable repeated result is not calibration.
Common questions
Does this replace screenshot comparison? This reviewed workflow concerns captured state and semantic claims. Pixel layout, colors and spacing require appropriate visual evidence.
Can it run offline? A fake provider checks wiring offline. Real Jev judgments require a provider request that sends the captured state. We have not installed or run this project.
Source: semantic-assert — README.md ↗
Explore these projects next
Continue reading
02 / JEV GUIDE
Jev API quickstart: send and validate your first request
Build a Jev API request with state, typed questions and Bearer authentication. Download a JSON fixture and review response checks.
Read guide →08 / JEV GUIDE
Jev browser automation with Playwright and Computer Use
Extract candidates from the DOM or accessibility tree, let Jev select an action, and execute through Playwright or Computer Use. Understand replay and stale-page limits.
Read guide →18 / JEV GUIDE
Jev writing quality checks for Markdown and technical docs
Turn writing requirements into line-level checks and document scores with slop-grader. Design custom rules, review false positives and preserve meaning during edits.
Read guide →