Bake-off: if unsure vs. if a fact is missing

TL;DR You cannot condition an agent rule on the model’s confidence, only on the presence of a fact: two prompts with the same intent both check clean at 0 error, 0 warning, 0 info, but the fact-conditioned one scores A (93/100) against B (88/100), takes deterministic asserts instead of a judge, and can be lifted onto the compiler plane, while IF you are unsure can do none of the three.

This is a bake-off, so the setup is the usual one: two prompts, the same intended behaviour, and the checker as referee. What makes this pair worth running is that the intended behaviour is not in dispute. Both files are trying to stop an agent cancelling the wrong order. They disagree about one thing only, which is what the rule is allowed to look at.

§1The rule everybody writes

Nobody sits down to write an unobservable trigger. It arrives by a very short path. The agent cancels the wrong order, somebody reads the transcript, and the obvious diagnosis is that the agent should have known it did not have enough to go on. So the fix is written in exactly those words, and it goes into the file below the rules that already work.

The sentence is fine English and terrible specification. Unsure is not a property of the conversation, it is a claim about the model’s internal state, and a language model is a poorly calibrated reporter of its own internal state. The rule does not get ignored. It gets answered, differently, every time, for reasons that are not written down anywhere.

Note that the adjective route is no better, and here at least the checker has something to say.

# Role
You are a support agent for Acme, an online retailer.

# Constraints
- You MUST be confident before you cancel an order.

§2The two contenders

Contender A conditions on the model. Three rules, each annotated, one $IMPORTed cancellation tool.

<$CONFIG>
  $IMPORT tool cancel_order
</$CONFIG>

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for Acme, an online retailer.

# Constraints
@@ ask_when_unsure: do not cancel the wrong order
- IF you are unsure which order the customer means THEN ask a clarifying question.
@@ ambiguity: a vague cancellation request needs detail
- You SHOULD ask for more detail when a request is ambiguous.
@@ cancel: the customer asked, so act
- You MUST call @[cancel_order] when the customer asks to cancel.

$TEST vague_cancel
  - input:: Please cancel my order.
  - expect::
    - You are unsure which order the customer means, so you ask a clarifying question.

Contender B conditions on the transcript. Same intent, same tool, same number of rules, and the word unsure does not appear. Instead the file names the thing whose absence is the actual problem.

<$CONFIG>
  $IMPORT tool cancel_order
</$CONFIG>

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for Acme, an online retailer.

# Constraints
@@ need_id: an order id is the only key the order system takes
- You MUST ask for an order id before you call @[cancel_order].
@@ one_question: a support turn that asks two things gets one answer
- You MUST ask at most 1 question in a reply.
@@ cancel: the id is present, so act
- WHEN the customer gives an order id THEN call @[cancel_order].

$TEST no_id
  - input:: Please cancel my order.
  - expect::
    - contains "order id"
    - at most 2 sentences

$TEST with_id
  - input:: Please cancel order A-4417.
  - expect::
    - contains "A-4417"
    - at most 2 sentences

§3Round 1: check ties, score picks a side

The referee’s first verdict is a tie, and an honest bake-off has to lead with that. TypeGlish reads the action a rule commands and largely leaves the guard alone, so an unobservable condition is not a diagnostic. Both files are clean, and --strict, which escalates every correctness advisory to a blocking error, escalates nothing on either one.

$ typeglish check confidence.tg fact.tg
✓ 2 files - 0 error, 0 warning, 0 info

$ typeglish check confidence.tg fact.tg --strict
✓ 2 files - 0 error, 0 warning, 0 info

The score is where the pair comes apart, and it comes apart on one facet.

confidence.tg - B (88/100)  proven errors: none  tiers: base+z3
  planes  runtime 84 (what the model reads) · hygiene 100 (source only)
  facets  enforceability 53 x.21 · hardness 83 x.12 · directness 100 x.08 · consistency 100 x.17
          structure 100 x.12 (hygiene) · annotation 100 x.12 (hygiene) · style 100 x.08 · security 100 x.08
  lever   enforceability 53/100 (up to +10 overall) - Write rules as MUST / NEVER <verb> … with concrete
          bounds ("at most 3 sentences"), not vague qualities - a rule the checker can parse is a rule it can defend.

fact.tg - A (93/100)  proven errors: none  tiers: base+z3
  planes  runtime 91 (what the model reads) · hygiene 100 (source only)
  facets  enforceability 68 x.21 · hardness 100 x.12 · directness 97 x.08 · consistency 100 x.17
          structure 100 x.12 (hygiene) · annotation 100 x.12 (hygiene) · style 100 x.08 · security 100 x.08

Enforceability is defined as the share of rule lines that lower into checkable intermediate representation. Contender A has three rules and one of them, the IF unsure line, contributes an action the checker can read behind a condition it cannot. That is the 53. The number is not a judgement about writing quality; it is a count of how much of your file the machine got to keep.

A green check says nothing was disproved. The score says how much was even readable.

§4Round 2: how many cases cover the rule?

Here is the question that settles it, and it is a question you can ask about any rule in any prompt, with no tooling at all: how many test cases does it take to cover this trigger?

For contender B the answer is two. The customer’s message either contains an order id or it does not, that is the entire domain, and two cases exhaust it. For contender A the answer is that nobody knows. To write the second case you would have to decide which inputs make a model unsure, and there is no such list. You can guess at an ambiguous message, but you cannot claim to have covered the trigger, because the trigger is not a set.

Run both suites offline and the shape of the difference shows up in the output.

✓ confidence.tg  coverage: 1/1 rules exercised
  · vague_cancel - "Please cancel my order." (not run)
      ✓ rubric 0.00 - not run (--dry)
✓ 1 prompt - 0 failed

✓ fact.tg  coverage: 1/2 rules exercised
  · no_id - "Please cancel my order." (not run)
      ✓ contains "order id"
      ✓ at most 2 sentences
  · with_id - "Please cancel order A-4417." (not run)
      ✓ contains "A-4417"
      ✓ at most 2 sentences
✓ 1 prompt - 0 failed

The JSON says it more bluntly, because the two instruments land in different fields. This is the same split we measured in deterministic asserts vs. an LLM judge for agent tests, and the reason it recurs here is not a coincidence: an expectation can only be an assert if there is a string in the reply to match, and a trigger can only be enumerated if it has members.

// confidence.tg
{ "id": "vague_cancel", "status": "skip",
  "asserts": [],
  "rubric": { "score": 0, "pass": true, "rationale": "not run (--dry)" } }

// fact.tg
{ "id": "no_id", "status": "skip",
  "asserts": [ { "detail": "contains \"order id\"", "pass": true },
               { "detail": "at most 2 sentences", "pass": true } ],
  "rubric": null }

§5Round 3: a fact can move to the compiler plane

The last round is the one that changes how you build, rather than how you write. A trigger with a closed domain is not stuck in prose. If the host runtime already knows the answer, the trigger can be promoted to a typed input, and then the branch is settled by the compiler before the model is involved at all.

In a real contact centre this is usually true. The conversation was opened from an order page, or it was not. The CRM has an account attached, or it does not. That is a fact the runtime holds, so declare it and switch on it.

<$CONFIG>
  $IMPORT tool cancel_order
  $REQUIRE variable order_context: one of known, missing
</$CONFIG>

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for Acme, an online retailer.

# Constraints
@@ one_question: a support turn that asks two things gets one answer
- You MUST ask at most 1 question in a reply.

$SWITCH ON @{order_context}
  - known:: You MUST call @[cancel_order] once the customer confirms a cancellation.
  - missing:: You MUST ask for an order id before you discuss a cancellation.

Build it twice and the point lands. The losing arm is not de-prioritised or out-weighed by a nearby rule. It is absent.

$ typeglish build lifted.tg --vars '{"order_context":"missing"}'
✓ built .typeglish/dist/lifted.txt ← lifted.tg (4101a17efd6a, full)

# Role
You are a support agent for Acme, an online retailer.

# Constraints
- You MUST ask at most 1 question in a reply.

You MUST ask for an order id before you discuss a cancellation.

$ typeglish build lifted.tg --vars '{"order_context":"known"}'
✓ built .typeglish/dist/lifted.txt ← lifted.tg (2a9fbbd58892, full)

# Role
You are a support agent for Acme, an online retailer.

# Constraints
- You MUST ask at most 1 question in a reply.

You MUST call cancel_order once the customer confirms a cancellation.

Now try to do the same thing to contender A. You would write $REQUIRE variable unsure: one of yes, no, and the compiler would accept it without complaint, because a checker cannot know that nothing in your stack can fill that field. The wall is not the language. It is that there is no caller who can honestly supply the value.

Two smaller findings from building this file, both worth having:

• The domain members started out as one of known, unknown and that draws a prompt/unregistered-doer info finding on the unknown arm. Renaming the member to missing clears it, with no other change. Enum members are read as English words, which is the same behaviour that bit the voicemail arm in building a ticket-triage agent, prompt-first.
• Contender B’s first rule, You MUST ask for an order id before you call @[cancel_order], cannot be caught contradicting a later NEVER ask for an order id, because the before clause scopes it into a different slot. Strip the clause and the same pair is 2 blocking logic/contradiction errors. A qualifier you added for precision is also a qualifier that hides the rule from the prover, which is item 1 on the add-a-rule checklist and still the easiest way to write a rule nobody can defend.

Do not ask the model how it feels. Ask the transcript what it contains.

§6The verdict

Contender B wins on three of the four instruments and ties on the fourth, which is the tie that matters most: check cannot tell them apart, so nothing in CI will ever stop contender A shipping. The rule reads well, it passes, and it works often enough that no single transcript indicts it.

The transferable move is small. When you catch yourself writing a guard about the agent’s state, stop and ask what fact you actually wanted it to have. Unsure which order means no order id. If the request is ambiguous usually means the message names no product, no order and no date. If you do not know the answer means the retrieved context returned nothing. In every case the second phrasing is longer, uglier, and checkable, and it is the one you can hand to a test.