Bake-off: deterministic asserts vs. an LLM judge for agent tests
Two suites for the same support prompt. Same case, same intent, same file: one written as asserts, one written as prose a model grades. They compile identically and score identically, and only one of them can turn a build red on its own.
TL;DR Give every agent test case at least one deterministic assert (contains "...", matches /.../, at most N sentences) so the suite can fail in CI with no model call and no API key, and keep prose expectations for the part only a reader can grade. typeglish test --dry --json tells you which is which: an assert lands in asserts, and anything judged collapses into rubric.
Everyone writing agent prompts eventually writes evals, and the first real decision is how to phrase an expectation. You can say what the reply must literally contain, or you can describe the reply you wanted and let a model decide whether you got it. The second reads better, and it is what most teams write, because it is how you would brief a colleague. This bake-off puts the two side by side on one support prompt and lets the toolchain referee. The tie is longer than you would expect, and the tiebreak is not about accuracy.
§1The two contenders
One prompt, four rules, one case: a customer asks for a price, and the agent is not allowed to quote one. Contender A writes the expectation as two deterministic asserts. Contender B writes the same expectation in the rules’ own words and leaves the grading to a judge. Nothing else differs.
<$CONFIG> $IMPORT tool open_ticket </$CONFIG> # Role @@ role: name the domain so "helpful" has a scope You are a support agent for Acme, a SaaS analytics company. # Constraints @@ no_prices: a quoted price goes stale and binds the company - You NEVER quote a price. @@ pricing_route: pricing intent belongs to the sales team - You MUST send every pricing question to sales. @@ brevity: three sentences keeps a support reply scannable - You MUST reply in at most 3 sentences. @@ ticket: a bug report is worthless without a ticket behind it - WHEN a customer reports a bug THEN you MUST call @[open_ticket]. $TEST pricing - input:: How much does the Pro plan cost per seat? - expect:: - contains "sales" - at most 3 sentences
<$CONFIG> $IMPORT tool open_ticket </$CONFIG> # Role @@ role: name the domain so "helpful" has a scope You are a support agent for Acme, a SaaS analytics company. # Constraints @@ no_prices: a quoted price goes stale and binds the company - You NEVER quote a price. @@ pricing_route: pricing intent belongs to the sales team - You MUST send every pricing question to sales. @@ brevity: three sentences keeps a support reply scannable - You MUST reply in at most 3 sentences. @@ ticket: a bug report is worthless without a ticket behind it - WHEN a customer reports a bug THEN you MUST call @[open_ticket]. $TEST pricing - input:: How much does the Pro plan cost per seat? - expect:: - sends the pricing question to sales - never quotes a price
Both are real code, not comments. The suite is part of the file the compiler reads, so a malformed case is a blocking error rather than a note nobody sees: write the expectation as a stray line and the build stops.
# Role @@ role: name the domain so "helpful" has a scope You are a support agent for Acme, a SaaS analytics company. # Constraints @@ no_prices: a quoted price goes stale and binds the company - You NEVER quote a price. $TEST pricing - input:: How much does the Pro plan cost per seat? The reply points at sales and quotes nothing.
$TEST body takes - input:: and - expect:: and nothing else. Drop the bullet and the expectation is not a loose expectation, it is a syntax error.badsuite.tg:9:1 error structure/bad-test only "- input::" and "- expect::" fields belong in $TEST pricing: "The reply points at sales and quotes nothing." is not one. ✗ 1 file - 1 error, 0 warning, 0 info
§2Round one: the toolchain calls it a tie
Run the two contenders through the checker and the scorer and there is nothing to separate them. Zero diagnostics each. Identical grades, facet for facet, including the two facets that read the source rather than the runtime.
✓ 2 files - 0 error, 0 warning, 0 info asserts.tg - A (100/100) proven errors: none tiers: base+z3 planes runtime 100 (what the model reads) · hygiene 100 (source only) facets enforceability 100 x.21 · hardness 100 x.12 · directness 98 x.08 · consistency 100 x.17 structure 100 x.12 (hygiene) · annotation 100 x.12 (hygiene) · style 100 x.08 · security 100 x.08 judged.tg - A (100/100) proven errors: none tiers: base+z3 planes runtime 100 (what the model reads) · hygiene 100 (source only) facets enforceability 100 x.21 · hardness 100 x.12 · directness 98 x.08 · consistency 100 x.17 structure 100 x.12 (hygiene) · annotation 100 x.12 (hygiene) · style 100 x.08 · security 100 x.08
$TEST block is control plane: it is graded out of band and never reaches the model, so it cannot move a facet that measures what the model reads.That tie is the reason this decision usually gets made by taste. Both files pass, both files look professional in review, and the prose version reads better out loud. If your gate is check plus a score floor, you will never hear about the difference. You hear about it the first time you want the suite to run somewhere that has no model.
§3Round two: which suite can fail on its own
There are three ways to run a suite, and the CLI is blunt about what each one costs. --dry is fully offline. A live run answers each case with a model and needs a key. --no-judge sits in between.
Test options: --dry fully offline: validate the suite + report rule coverage, zero model calls --no-judge deterministic asserts only - skip the judge on prose expectations (still generates) --model <id> the model that answers each case (and judges, unless --no-judge)
--no-judge closely. It keeps the deterministic asserts and drops the judge. A suite with no deterministic asserts has nothing left when you use it.Now run both contenders offline. Same command, same flags, and the first honest difference in the whole bake-off.
✓ asserts.tg coverage: 1/4 rules exercised · pricing - "How much does the Pro plan cost per seat?" (not run) ✓ contains "sales" ✓ at most 3 sentences ✓ judged.tg coverage: 4/6 rules exercised · pricing - "How much does the Pro plan cost per seat?" (not run) ✓ rubric 0.00 - not run (--dry)
The JSON is where it stops being a matter of reading style and becomes a data structure. Every expectation lands in exactly one of two places: asserts, which is a list of mechanical checks, or rubric, which is a single judged verdict for the whole case.
"cases": [
{
"id": "pricing",
"status": "skip",
"asserts": [
{ "detail": "contains \"sales\"", "pass": true },
{ "detail": "at most 3 sentences", "pass": true }
],
"rubric": null
}
],
"coverage": { "covered": 1, "total": 4 }
"rubric": null is the sentence that matters: nothing in this case needs a model to have an opinion. Offline, the asserts report pass: true because there is no reply to fail against yet, but the shape is already fixed."cases": [
{
"id": "pricing",
"status": "skip",
"asserts": [],
"rubric": { "score": 0, "pass": true, "rationale": "not run (--dry)" }
}
],
"coverage": { "covered": 4, "total": 6 }
asserts array is a suite that cannot produce a verdict without a judge. That is not a bug in contender B, it is contender B's actual contract, now written down where CI can read it.Contender B does win something here, and it is not a consolation prize. Coverage attributes a case to the rules its expectations name, and prose written in the rules’ own words attributes far better: never quotes a price lands on the no_prices rule, while contains "sales" is a string that names no rule at all. The denominator moves too, because each prose expectation is itself a statement in the file, which is why contender B reports out of 6 rather than out of 4. Treat the number as a map rather than a gate, the way delete a line, see who notices treats it: it tells you which rules no case is watching.
A suite that cannot fail without a model is documentation with a pass rate.
§4Round three: two things an assert cannot do
The obvious limit is expressiveness, and it is real. No regex tells you whether a refusal was gracious, whether the reply answered the question the customer actually asked, or whether "we can help with that" was followed by help. Those are readings, and a judge is the only instrument that performs a reading. That is the honest case for prose expectations, and it is why the rest of this post is not an argument for deleting them.
The less obvious limit is that the assert forms are a closed list, and the negations are not on it. contains "...", matches /.../, at most N sentences, at least N sentences and at most N words are read as asserts. Write the same idea in the negative, which is exactly how a guardrail comes out of your fingers, and it quietly becomes a judged line.
# Role @@ role: name the domain so "helpful" has a scope You are a support agent for Acme, a SaaS analytics company. # Constraints @@ no_prices: a quoted price goes stale and binds the company - You NEVER quote a price. @@ pricing_route: pricing intent belongs to the sales team - You MUST send every pricing question to sales. $TEST pricing - input:: How much does the Pro plan cost per seat? - expect:: - does not contain "$"
✓ negative.tg coverage: 0/2 rules exercised · pricing - "How much does the Pro plan cost per seat?" (not run) ✓ rubric 0.00 - not run (--dry) ✓ 1 prompt - 0 failed
does not contain "$" reads like a mechanical check and is graded as prose. If you meant it mechanically, say it as a pattern: matches /^[^$]*$/ is an assert, and it is the deterministic way to say "no dollar figure in this reply".And one more boundary worth knowing, because it looks like a hole and is really a division of labor: the checker does not reconcile an assert against the rule it is testing. A case that asserts at most 5 sentences, under a prompt whose rule says at most 3 sentences, compiles clean.
# Role @@ role: name the domain so "helpful" has a scope You are a support agent for Acme, a SaaS analytics company. # Constraints @@ brevity: three sentences keeps a support reply scannable - You MUST reply in at most 3 sentences. $TEST brevity - input:: How much does the Pro plan cost per seat? - expect:: - at most 5 sentences
at most 3 sentences against at least 5 sentences as two rules and it blocks the build with logic/numeric, because no reply satisfies both. An assert is a claim about a reply rather than about a rule, so there is nothing on the other side of it to prove against. Test bounds are yours to keep honest; the arithmetic in your prompt is about the numbers the prover does read.§5The verdict: one expect list, both instruments
Nobody wins this outright, which is the useful outcome, because the two styles are answering different questions. The assert answers can this suite fail by itself. The judge answers was the reply any good. A case that only asserts is cheap and blind; a case that only describes is perceptive and mute in CI. So write both, in one - expect:: list, and let the runner sort them.
<$CONFIG> $IMPORT tool open_ticket </$CONFIG> # Role @@ role: name the domain so "helpful" has a scope You are a support agent for Acme, a SaaS analytics company. # Constraints @@ no_prices: a quoted price goes stale and binds the company - You NEVER quote a price. @@ pricing_route: pricing intent belongs to the sales team - You MUST send every pricing question to sales. @@ brevity: three sentences keeps a support reply scannable - You MUST reply in at most 3 sentences. @@ ticket: a bug report is worthless without a ticket behind it - WHEN a customer reports a bug THEN you MUST call @[open_ticket]. $TEST pricing - input:: How much does the Pro plan cost per seat? - expect:: - matches /^[^$]*$/ - at most 3 sentences - sends the pricing question to sales $TEST bug_report - input:: The dashboard has been blank since your last deploy. - expect:: - at most 3 sentences - opens a ticket for the bug
✓ support.tg coverage: 4/6 rules exercised · pricing - "How much does the Pro plan cost per seat?" (not run) ✓ matches /^[^$]*$/ ✓ at most 3 sentences ✓ rubric 0.00 - not run (--dry) · bug_report - "The dashboard has been blank since your last deploy." (not run) ✓ at most 3 sentences ✓ rubric 0.00 - not run (--dry) ✓ 1 prompt - 0 failed
The practical rule fits on one line: if you can say it mechanically, say it mechanically, and then also say what you meant. Pick the assert for the part of the rule that has an edge, a number, a required word, a forbidden pattern. Pick the prose for the judgment you would otherwise leave to a code review nobody schedules. Then check the JSON once, before you trust the suite, because the difference between an assert and a judged line is invisible in the file and explicit in --dry --json.
Assert the part with an edge. Judge the part with a reading. Never ship a case with neither.
§6Common questions
- How do I test an AI agent's system prompt without calling a model?
- Write the cases into the prompt file as
$TESTblocks and runtypeglish test --dry. It is fully offline: it validates every case, reports which of your rules the cases exercise, and spends zero model calls, so it can run on every commit. What it cannot do is grade a reply, because there is no reply until something answers the case. A malformed suite fails there too: a stray line inside a$TESTisstructure/bad-test, a blocking error. - Should I use assertions or an LLM judge for prompt evals?
- Both, in the same case. Give every case at least one deterministic assert so the suite can fail without a judge and without an API key, and use prose expectations for the part of the behavior only a reader can grade, like whether a refusal was polite or whether the reply actually answered the question. A suite made only of judged prose has no verdict at all when the judge is switched off.
- Why is my test expectation being graded by a judge instead of checked?
- Because it is not one of the deterministic assert forms.
contains "...",matches /.../,at most N sentences,at least N sentencesandat most N wordsare read as asserts; anything else, including the negated forms likedoes not contain "$", is prose and goes to the judge. Runtypeglish test --dry --jsonand look at each case: an assert lands in theassertsarray, and everything judged collapses intorubric. If you want a negative checked deterministically, write it as a regex, for examplematches /^[^$]*$/for a reply with no dollar sign in it. - Does the checker verify my test expectations against my rules?
- No, and it is worth knowing where that line sits. An assert is a claim about a reply, not a claim about a rule, so a case that asserts
at most 5 sentencesunder a prompt whose rule saysat most 3 sentencescompiles clean and reports no conflict. Contradictions are proven between rules; a loose assert is only caught by reading it, or by a live run against a model that obeys the stricter rule.
The $TEST block is control plane, graded out of band and never sent to the model, which is why neither contender could move the score: the suite is source, not prompt. That is the same seam as your system prompt has a compile time. For the case that starts a suite rather than finishes one, the failing transcript you write down before you diagnose anything, see step one of your agent said the wrong thing, now what?, and for a case pinning a money rule in a live build, building a refund agent, prompt-first.