Bake-off: one test per rule vs. one test per contact
There are two ways to write the test suite for a support agent: one case per rule, or one case per realistic customer message. On the same prompt they tie on coverage and on the grade, and only one of them finds the question your rules never answered.
TL;DR Write one $TEST per rule, with an assert on the word only that rule produces, and then add one case per real customer message that can fire two rules at once: the per-rule suite keeps coverage honest, and the per-contact suite is the only one that finds the collisions, like a customer who reports an outage and asks to cancel in the same sentence.
Corvid Broadband runs a chat agent on its billing queue. The prompt is short and reviewed, and it has no tests, which the team lead wants fixed before the next release. Two people pick it up. One writes a case for every rule, the way you would unit-test a function. The other pulls a week of real transcripts and writes a case for each kind of contact the desk actually gets. Both suites report full coverage. Here is what each one knows that the other does not.
§1The prompt under test
Five rules: a length cap, a card-number ban, a permission to waive a late fee, and two transfers to different teams.
# Role @@ role: one sentence names the speaker, so every later "you" is the agent You are Sable, the billing support agent for Corvid Broadband. # Constraints @@ brevity: three sentences keeps a chat reply scannable - MUST keep every reply to at most 3 sentences. @@ no_card: a card number in chat is a PCI incident - NEVER ask for a card number. @@ late_fee: the desk may waive one late fee, never two - WHEN a customer disputes a late fee THEN MAY waive one late fee. @@ outage: an outage credit is the network team's decision - WHEN a customer reports an outage THEN MUST transfer them to the network team. @@ cancel: cancellation is a retention call - WHEN a customer asks to cancel THEN MUST transfer them to the retention team.
A (96/100), all five rule-shaped lines read. The one lever score names is the MAY on L11, and it is a permission by design, so it stays.§2Two suites, same numbers
Suite A is one case per rule. Each input is short and does one thing, and each expectation borrows the wording of the rule it is aimed at.
$TEST brevity - input:: Why is my bill higher this month? - expect:: - at most 3 sentences $TEST no_card - input:: I want to pay my balance now. - expect:: - never asks for a card number - matches /^(?!.*card number)/ $TEST late_fee - input:: I was charged a late fee but I paid on time. - expect:: - waives one late fee $TEST outage - input:: My internet has been down since last night. - expect:: - transfers them to the network team - contains "network team" $TEST cancel - input:: I want to cancel my contract. - expect:: - transfers them to the retention team - contains "retention team"
Suite B is one case per contact. The inputs are written the way customers actually write: two things at once, a bit of frustration, no tidy separation of intents. That is the reality your router assumes one thing is wrong is about, turned into test input.
$TEST outage_and_cancel - input:: My internet has been down for three days and honestly I just want to cancel. - expect:: - transfers them to the network team - contains "network team" - at most 3 sentences $TEST late_fee_and_payment - input:: You charged me a late fee even though I paid on time, and I want to pay the rest right now. - expect:: - waives one late fee - never asks for a card number - matches /^(?!.*card number)/
$ npx typeglish score per-rule.tg per-rule.tg — A (96/100) proven errors: none tiers: base+z3 rules 5 of 5 rule-shaped lines read · 4/4 covered by $TEST $ npx typeglish test per-rule.tg --dry ✓ per-rule.tg coverage: 4/4 rules exercised $ npx typeglish score per-contact.tg per-contact.tg — A (96/100) proven errors: none tiers: base+z3 rules 5 of 5 rule-shaped lines read · 4/4 covered by $TEST $ npx typeglish test per-contact.tg --dry ✓ per-contact.tg coverage: 4/4 rules exercised
base.tg, the same 4/4. Tests are control plane, so the grade was never going to move, which is the finding of delete every test. The more interesting question is why the denominator is 4 when the prompt has five rules, and what each suite did to reach it.§3What the 4 counts
The denominator first. The permission on L11 is not in it: a MAY requires nothing, so there is nothing for a case to fail on. The four counted rules are the length cap, the card ban and the two transfers.
Now the numerator. Suite B never says the word retention in an expectation, yet it covers the retention rule. Coverage is a wording match, inflection-tolerant: a case covers a rule when its expect:: line shares the rule's words. Both transfer rules share almost all of theirs. One case is enough to show it.
$TEST outage - input:: My internet has been down since last night. - expect:: - transfers them to the network team $ npx typeglish test one-case.tg --dry ✓ one-case.tg coverage: 2/4 rules exercised uncovered (no case shares the rule's wording): L7 keep every reply to at most 3 sentences. L9 ask for a card number. · outage — "My internet has been down since last night." (not run) · rubric — 1 prose expectation, judged on a live run (needs ANTHROPIC_API_KEY)
So the two 4/4s are not the same claim. Suite A reaches it with a case aimed at each rule, and its contains "retention team" assert is the line that actually distinguishes one transfer from the other. Suite B reaches it partly by resemblance. The same property turned up in your agent works nights, where coverage matched the wording of a rule rather than the arm it sat in. The habit that follows is small: in a per-rule case, assert on the word that only that rule produces, usually the destination or the tool, and read the uncovered list rather than the fraction.
That is suite A's win. It makes the coverage number mean what it says.
§4The question suite A cannot ask
Look at the first input in suite B again: my internet has been down for three days and honestly I just want to cancel. Both guards match. The outage rule says transfer to the network team; the cancel rule says transfer to the retention team. The prompt has no opinion about which wins, and suite A will never notice, because no input in it carries two intents.
The person writing suite B had to put something in expect::, and the only way to do that was to decide. The desk's answer is the network team: nobody should get a retention offer while their line is still down. So the case gets a negative assert.
$TEST outage_and_cancel - input:: My internet has been down for three days and honestly I just want to cancel. - expect:: - transfers them to the network team - contains "network team" - matches /^(?!.*retention)/ - at most 3 sentences $ npx typeglish check contact-noexc.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish test contact-noexc.tg --dry ✓ contact-noexc.tg coverage: 4/4 rules exercised
That is worth being clear about. A $TEST is control plane: check validates its shape and test --dry reports what it covers, but no step reconciles what a case expects with what the rules require. The contradiction between the test and L15 will surface on the first live run, as a failure that looks like the model misbehaving, when the model is doing what the prompt told it to. A decision that lives only in a test is a decision the model has never been told.
That is suite B's win, and it is the bigger one. The per-contact case found a missing rule. It did it by forcing a human to answer a question the prompt had left open.
§5The verdict: keep both, and write the answer twice
The fix goes in the rules first and the test second. The precedence becomes a rule with its own guard, and the file keeps both kinds of case, grouped and commented so a later tidy-up does not delete the contact cases as duplicates.
# Role @@ role: one sentence names the speaker, so every later "you" is the agent You are Sable, the billing support agent for Corvid Broadband. # Constraints @@ brevity: three sentences keeps a chat reply scannable - MUST keep every reply to at most 3 sentences. @@ no_card: a card number in chat is a PCI incident - NEVER ask for a card number. @@ late_fee: the desk may waive one late fee, never two - WHEN a customer disputes a late fee THEN MAY waive one late fee. @@ outage: an outage credit is the network team's decision - WHEN a customer reports an outage THEN MUST transfer them to the network team. @@ cancel: cancellation is a retention call - WHEN a customer asks to cancel THEN MUST transfer them to the retention team. @@ outage_first: nobody gets a save offer while their line is still down - WHEN a customer reports an outage and asks to cancel THEN NEVER transfer them to the retention team. // one case per rule: the assert names what only that rule produces $TEST outage - input:: My internet has been down since last night. - expect:: - transfers them to the network team - contains "network team" $TEST cancel - input:: I want to cancel my contract. - expect:: - transfers them to the retention team - contains "retention team" $TEST no_card - input:: I want to pay my balance now. - expect:: - never asks for a card number - matches /^(?!.*card number)/ // one case per real contact: the collisions no single-rule case can reach $TEST outage_and_cancel - input:: My internet has been down for three days and honestly I just want to cancel. - expect:: - transfers them to the network team - contains "network team" - matches /^(?!.*retention)/ - at most 3 sentences $TEST late_fee_and_payment - input:: You charged me a late fee even though I paid on time, and I want to pay the rest right now. - expect:: - waives one late fee - never asks for a card number - matches /^(?!.*card number)/ - at most 3 sentences
$ npx typeglish check final.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish score final.tg final.tg — A (97/100) proven errors: none tiers: base+z3 planes runtime 96 (what the model reads) · hygiene 100 (source only) rules 6 of 6 rule-shaped lines read · 5/5 covered by $TEST $ npx typeglish test final.tg --dry ✓ final.tg coverage: 5/5 rules exercised · outage_and_cancel — "My internet has been down for three days and honestly I just" (not run) ✓ contains "network team" ✓ matches /^(?!.*retention)/ ✓ at most 3 sentences · rubric — 1 prose expectation, judged on a live run (needs ANTHROPIC_API_KEY)
One honest limit on the new rule. The checker passes the pair of cancel rules clean, but it does not prove that the narrower guard wins: it reads each guard as its own condition. Nothing in check would object if the exception were deleted tomorrow. The outage_and_cancel case is what holds it in place, which is exactly why the decision needs to be written in both places: the rule tells the model, the case tells the next editor.
The scorecard, then. Suite A wins on the thing the tools can measure: a case per rule, a distinguishing assert each, and a coverage number you can trust. Suite B wins on the thing they cannot: it is the only way two rules ever fire together inside the test suite, and that is where the gaps in a support prompt actually are. Five cases total here, and the two contact cases did the most work.
§6Common questions
- How many test cases should an AI support agent prompt have?
- One per binding rule, each with a deterministic assert on the word only that rule produces, plus one for every pair of rules a single customer message can trigger at once. The first set makes the coverage number honest; the second set is where the precedence questions live. For the billing prompt here that was five cases, and the two contact cases were the ones that found a missing rule.
- Why does my prompt show full test coverage when a rule is untested?
- Because coverage matches wording, not behaviour. A
$TESTcovers a rule when itsexpect::line shares the rule's words, so one expectation that saystransfers them to the network teamalso covers a rule that transfers them to the retention team. Intest --dryhere, that single case marked both transfer rules covered. Name the destination in acontainsassert and read the uncovered list, not just the fraction. - Should I write agent test cases from real customer messages or from the rules?
- Both, for different jobs. A case written from a rule has a short, single-purpose input and proves that rule in isolation. A case written from a real contact carries two or three intents in one message, the way customers actually write, and it is the only kind that makes two rules fire together. Keep them in the same file, grouped and commented, so nobody deletes the contact cases as duplicates.
- Does TypeGlish check that my tests agree with my rules?
- No. A
$TESTis control plane:checkvalidates its shape andtest --dryreports coverage, but nothing compares an expectation with the rules. In this bake-off a case asserting the agent never mentions retention sat beside a rule requiring a transfer to the retention team, and the file checked clean with full coverage. The disagreement only surfaces on a live run, which is why the fix belongs in the rules as well as the test.
The per-contact cases are cheap to write if you already have transcripts, and most contact centres have more than they can read. The trick is choosing which ones. Sort a week of contacts by how many intents the customer raised, not by volume, and start at the top: the message that mentions a fee, a fault and a cancellation in two sentences is the one your rules were never written against, because each rule was written by someone thinking about one situation. Every time a contact case forces a decision, write the decision as a rule in the prompt before you write it as an assert in the test. The same instinct, applied to guards rather than tests, is in every WHEN is a classifier: a guard is a label the model assigns to a message, and one message can earn two labels.