> Markdown rendition of https://typeglish.dev/blog/tests-per-rule-vs-per-contact ("Bake-off: one test per rule vs. one test per contact - Tidelines"). Canonical page: https://typeglish.dev/blog/tests-per-rule-vs-per-contact · All pages: https://typeglish.dev/llms.txt

[← Tidelines](https://typeglish.dev/blog)/ Best practices Sep 29, 2026

# Bake-off: one test per rule vs. one test per contact

There are two ways to write the test suite for a support agent: one case per rule, or one case per realistic customer message. On the same prompt they tie on coverage and on the grade, and only one of them finds the question your rules never answered.

by **TypeGlish team** 8 min read #best-practices

Same coverage. Different questions.

**TL;DR** Write one `$TEST` per rule, with an assert on the word only that rule produces, and then add one case per real customer message that can fire two rules at once: the per-rule suite keeps coverage honest, and the per-contact suite is the only one that finds the collisions, like a customer who reports an outage and asks to cancel in the same sentence.

Corvid Broadband runs a chat agent on its billing queue. The prompt is short and reviewed, and it has no tests, which the team lead wants fixed before the next release. Two people pick it up. One writes a case for every rule, the way you would unit-test a function. The other pulls a week of real transcripts and writes a case for each kind of contact the desk actually gets. Both suites report full coverage. Here is what each one knows that the other does not.

## §1 The prompt under test

Five rules: a length cap, a card-number ban, a permission to waive a late fee, and two transfers to different teams.

base.tg - a billing support agent, no tests yet ✓ A (96/100)

```typeglish
# Role
@@ role: one sentence names the speaker, so every later "you" is the agent
You are Sable, the billing support agent for Corvid Broadband.

# Constraints
@@ brevity: three sentences keeps a chat reply scannable
- MUST keep every reply to at most 3 sentences.
@@ no_card: a card number in chat is a PCI incident
- NEVER ask for a card number.
@@ late_fee: the desk may waive one late fee, never two
- WHEN a customer disputes a late fee THEN MAY waive one late fee.
@@ outage: an outage credit is the network team's decision
- WHEN a customer reports an outage THEN MUST transfer them to the network team.
@@ cancel: cancellation is a retention call
- WHEN a customer asks to cancel THEN MUST transfer them to the retention team.
```

Checks clean at `A (96/100)`, all five rule-shaped lines read. The one lever `score` names is the `MAY` on L11, and it is a permission by design, so it stays.

## §2 Two suites, same numbers

Suite A is one case per rule. Each input is short and does one thing, and each expectation borrows the wording of the rule it is aimed at.

per-rule.tg - suite A, appended to base.tg ✓ 4/4 rules exercised

```
$TEST brevity
  - input:: Why is my bill higher this month?
  - expect::
    - at most 3 sentences

$TEST no_card
  - input:: I want to pay my balance now.
  - expect::
    - never asks for a card number
    - matches /^(?!.*card number)/

$TEST late_fee
  - input:: I was charged a late fee but I paid on time.
  - expect::
    - waives one late fee

$TEST outage
  - input:: My internet has been down since last night.
  - expect::
    - transfers them to the network team
    - contains "network team"

$TEST cancel
  - input:: I want to cancel my contract.
  - expect::
    - transfers them to the retention team
    - contains "retention team"
```

Suite B is one case per contact. The inputs are written the way customers actually write: two things at once, a bit of frustration, no tidy separation of intents. That is the reality [your router assumes one thing is wrong](https://typeglish.dev/blog/multi-intent-contact) is about, turned into test input.

per-contact.tg - suite B, appended to base.tg ✓ 4/4 rules exercised

```
$TEST outage_and_cancel
  - input:: My internet has been down for three days and honestly I just want to cancel.
  - expect::
    - transfers them to the network team
    - contains "network team"
    - at most 3 sentences

$TEST late_fee_and_payment
  - input:: You charged me a late fee even though I paid on time, and I want to pay the rest right now.
  - expect::
    - waives one late fee
    - never asks for a card number
    - matches /^(?!.*card number)/
```

tg score and test --dry - both suites

```
$ npx typeglish score per-rule.tg
per-rule.tg — A (96/100)  proven errors: none  tiers: base+z3
  rules   5 of 5 rule-shaped lines read · 4/4 covered by $TEST

$ npx typeglish test per-rule.tg --dry
✓ per-rule.tg  coverage: 4/4 rules exercised

$ npx typeglish score per-contact.tg
per-contact.tg — A (96/100)  proven errors: none  tiers: base+z3
  rules   5 of 5 rule-shaped lines read · 4/4 covered by $TEST

$ npx typeglish test per-contact.tg --dry
✓ per-contact.tg  coverage: 4/4 rules exercised
```

Five cases against two, and the toolchain cannot tell them apart: the same grade as the untested `base.tg`, the same `4/4`. Tests are control plane, so the grade was never going to move, which is the finding of [delete every test](https://typeglish.dev/blog/test-suite-ablation). The more interesting question is why the denominator is 4 when the prompt has five rules, and what each suite did to reach it.

## §3 What the 4 counts

The denominator first. The permission on L11 is not in it: a `MAY` requires nothing, so there is nothing for a case to fail on. The four counted rules are the length cap, the card ban and the two transfers.

Now the numerator. Suite B never says the word *retention* in an expectation, yet it covers the retention rule. Coverage is a wording match, inflection-tolerant: a case covers a rule when its `expect::` line shares the rule's words. Both transfer rules share almost all of theirs. One case is enough to show it.

one-case.tg - base.tg plus a single outage case ✗ 2/4, both transfers counted

```
$TEST outage
  - input:: My internet has been down since last night.
  - expect::
    - transfers them to the network team

$ npx typeglish test one-case.tg --dry
✓ one-case.tg  coverage: 2/4 rules exercised
  uncovered (no case shares the rule's wording):
    L7  keep every reply to at most 3 sentences.
    L9  ask for a card number.
  · outage — "My internet has been down since last night." (not run)
      · rubric — 1 prose expectation, judged on a live run (needs ANTHROPIC_API_KEY)
```

L13 and L15 are both missing from the uncovered list. A case that expects the network team has, as far as coverage is concerned, tested the retention team too.

So the two `4/4`s are not the same claim. Suite A reaches it with a case aimed at each rule, and its `contains "retention team"` assert is the line that actually distinguishes one transfer from the other. Suite B reaches it partly by resemblance. The same property turned up in [your agent works nights](https://typeglish.dev/blog/agent-after-hours), where coverage matched the wording of a rule rather than the arm it sat in. The habit that follows is small: in a per-rule case, assert on the word that only that rule produces, usually the destination or the tool, and read the uncovered list rather than the fraction.

That is suite A's win. It makes the coverage number mean what it says.

## §4 The question suite A cannot ask

Look at the first input in suite B again: *my internet has been down for three days and honestly I just want to cancel.* Both guards match. The outage rule says transfer to the network team; the cancel rule says transfer to the retention team. The prompt has no opinion about which wins, and suite A will never notice, because no input in it carries two intents.

The person writing suite B had to put something in `expect::`, and the only way to do that was to decide. The desk's answer is the network team: nobody should get a retention offer while their line is still down. So the case gets a negative assert.

contact-noexc.tg - the decision written into the test, and only the test ✓ clean, 4/4

```
$TEST outage_and_cancel
  - input:: My internet has been down for three days and honestly I just want to cancel.
  - expect::
    - transfers them to the network team
    - contains "network team"
    - matches /^(?!.*retention)/
    - at most 3 sentences

$ npx typeglish check contact-noexc.tg
✓ 1 file — 0 error, 0 warning, 0 info

$ npx typeglish test contact-noexc.tg --dry
✓ contact-noexc.tg  coverage: 4/4 rules exercised
```

The case now says *never mention retention* while L15 still says *MUST transfer them to the retention team*, on an input that satisfies its guard. Clean check, full coverage. Nothing compares a test with a rule.

That is worth being clear about. A `$TEST` is control plane: `check` validates its shape and `test --dry` reports what it covers, but no step reconciles what a case expects with what the rules require. The contradiction between the test and L15 will surface on the first live run, as a failure that looks like the model misbehaving, when the model is doing what the prompt told it to. A decision that lives only in a test is a decision the model has never been told.

That is suite B's win, and it is the bigger one. The per-contact case found a missing rule. It did it by forcing a human to answer a question the prompt had left open.

## §5 The verdict: keep both, and write the answer twice

The fix goes in the rules first and the test second. The precedence becomes a rule with its own guard, and the file keeps both kinds of case, grouped and commented so a later tidy-up does not delete the contact cases as duplicates.

final.tg - the exception as a rule, and both kinds of case ✓ A (97/100)

```typeglish
# Role
@@ role: one sentence names the speaker, so every later "you" is the agent
You are Sable, the billing support agent for Corvid Broadband.

# Constraints
@@ brevity: three sentences keeps a chat reply scannable
- MUST keep every reply to at most 3 sentences.
@@ no_card: a card number in chat is a PCI incident
- NEVER ask for a card number.
@@ late_fee: the desk may waive one late fee, never two
- WHEN a customer disputes a late fee THEN MAY waive one late fee.
@@ outage: an outage credit is the network team's decision
- WHEN a customer reports an outage THEN MUST transfer them to the network team.
@@ cancel: cancellation is a retention call
- WHEN a customer asks to cancel THEN MUST transfer them to the retention team.
@@ outage_first: nobody gets a save offer while their line is still down
- WHEN a customer reports an outage and asks to cancel THEN NEVER transfer them to the retention team.

// one case per rule: the assert names what only that rule produces
$TEST outage
  - input:: My internet has been down since last night.
  - expect::
    - transfers them to the network team
    - contains "network team"

$TEST cancel
  - input:: I want to cancel my contract.
  - expect::
    - transfers them to the retention team
    - contains "retention team"

$TEST no_card
  - input:: I want to pay my balance now.
  - expect::
    - never asks for a card number
    - matches /^(?!.*card number)/

// one case per real contact: the collisions no single-rule case can reach
$TEST outage_and_cancel
  - input:: My internet has been down for three days and honestly I just want to cancel.
  - expect::
    - transfers them to the network team
    - contains "network team"
    - matches /^(?!.*retention)/
    - at most 3 sentences

$TEST late_fee_and_payment
  - input:: You charged me a late fee even though I paid on time, and I want to pay the rest right now.
  - expect::
    - waives one late fee
    - never asks for a card number
    - matches /^(?!.*card number)/
    - at most 3 sentences
```

tg check, score and test --dry - final.tg

```
$ npx typeglish check final.tg
✓ 1 file — 0 error, 0 warning, 0 info

$ npx typeglish score final.tg
final.tg — A (97/100)  proven errors: none  tiers: base+z3
  planes  runtime 96 (what the model reads) · hygiene 100 (source only)
  rules   6 of 6 rule-shaped lines read · 5/5 covered by $TEST

$ npx typeglish test final.tg --dry
✓ final.tg  coverage: 5/5 rules exercised
  · outage_and_cancel — "My internet has been down for three days and honestly I just" (not run)
      ✓ contains "network team"
      ✓ matches /^(?!.*retention)/
      ✓ at most 3 sentences
      · rubric — 1 prose expectation, judged on a live run (needs ANTHROPIC_API_KEY)
```

Six rules read, five counted, five covered. In a dry run the ticks mean each assert parsed as a deterministic check; they run against the actual reply on a live run, with no judge needed, which is the argument of [deterministic asserts vs. an LLM judge](https://typeglish.dev/blog/test-asserts-vs-judge).

One honest limit on the new rule. The checker passes the pair of cancel rules clean, but it does not prove that the narrower guard wins: it reads each guard as its own condition. Nothing in `check` would object if the exception were deleted tomorrow. The `outage_and_cancel` case is what holds it in place, which is exactly why the decision needs to be written in both places: the rule tells the model, the case tells the next editor.

The scorecard, then. Suite A wins on the thing the tools can measure: a case per rule, a distinguishing assert each, and a coverage number you can trust. Suite B wins on the thing they cannot: it is the only way two rules ever fire together inside the test suite, and that is where the gaps in a support prompt actually are. Five cases total here, and the two contact cases did the most work.

## §6 Common questions

**How many test cases should an AI support agent prompt have?**

One per binding rule, each with a deterministic assert on the word only that rule produces, plus one for every pair of rules a single customer message can trigger at once. The first set makes the coverage number honest; the second set is where the precedence questions live. For the billing prompt here that was five cases, and the two contact cases were the ones that found a missing rule.

**Why does my prompt show full test coverage when a rule is untested?**

Because coverage matches wording, not behaviour. A `$TEST` covers a rule when its `expect::` line shares the rule's words, so one expectation that says `transfers them to the network team` also covers a rule that transfers them to the retention team. In `test --dry` here, that single case marked both transfer rules covered. Name the destination in a `contains` assert and read the uncovered list, not just the fraction.

**Should I write agent test cases from real customer messages or from the rules?**

Both, for different jobs. A case written from a rule has a short, single-purpose input and proves that rule in isolation. A case written from a real contact carries two or three intents in one message, the way customers actually write, and it is the only kind that makes two rules fire together. Keep them in the same file, grouped and commented, so nobody deletes the contact cases as duplicates.

**Does TypeGlish check that my tests agree with my rules?**

No. A `$TEST` is control plane: `check` validates its shape and `test --dry` reports coverage, but nothing compares an expectation with the rules. In this bake-off a case asserting the agent never mentions retention sat beside a rule requiring a transfer to the retention team, and the file checked clean with full coverage. The disagreement only surfaces on a live run, which is why the fix belongs in the rules as well as the test.

Field note

The per-contact cases are cheap to write if you already have transcripts, and most contact centres have more than they can read. The trick is choosing which ones. Sort a week of contacts by how many intents the customer raised, not by volume, and start at the top: the message that mentions a fee, a fault and a cancellation in two sentences is the one your rules were never written against, because each rule was written by someone thinking about one situation. Every time a contact case forces a decision, write the decision as a rule in the prompt before you write it as an assert in the test. The same instinct, applied to guards rather than tests, is in [every WHEN is a classifier](https://typeglish.dev/blog/every-when-is-a-classifier): a guard is a label the model assigns to a message, and one message can earn two labels.

∿ washed up Sep 29, 2026 ∿