Rewrite: a five really helps the team
The wrap-up section of a support prompt is where the CSAT target leaks in: ask for a rating, skip the angry ones, mention that a five helps. It checks clean, and it turns your satisfaction score into a measure of who the agent chose to ask.
TL;DR Take the rating ask out of the agent: an agent that asks for a score, skips customers who seem annoyed, and mentions that a five helps is choosing its own sample and lobbying its own judge, so let the platform send the survey after every chat, write NEVER ask for a rating, and route a customer who says the chat failed them to a person.
Every support prompt that has lived through a CSAT target has a section like this one near the bottom. Nobody wrote it to game anything. It was written by people who wanted more survey responses and fewer angry ones, and it reads as reasonable until you ask what the number it produces actually measures.
§1The before
Orrin Pet Supplies sells food, litter and flea treatments online, and its chat agent is good at the middle of a conversation. The end is a different story. The wrapping-up section was written the quarter the support team got a CSAT target, and every line of it is somebody trying to help the number.
# Role You are the chat support agent for Orrin Pet Supplies, an online pet food and supplies shop. # Wrapping up - ALWAYS ask for a rating from 1 to 5. - IF the customer seems annoyed THEN NEVER ask for a rating. - Try to get a 5 where possible. - MUST mention that a 5 helps the team. - NEVER pressure the customer. - WHEN the customer gives a low rating THEN MUST apologize.
$ typeglish check rating.tg rating.tg:7:3 info prompt/hedging Hedging - "Try to" turns this instruction into a suggestion the model may skip. Delete the hedge, or commit to a modal (MUST / NEVER / SHOULD). rating.tg:7:18 info prompt/hedging Hedging - "where possible" turns this instruction into a suggestion the model may skip. Delete the hedge, or commit to a modal (MUST / NEVER / SHOULD). rating.tg:7:1 info prompt/unparsed-rule Reads like a rule, but the rule reader cannot parse it as one, so it earns no Enforceability credit and no measurability check: no modal (MUST / NEVER / SHOULD) and "Try" is not a verb the checker knows, so it reads as prose. Fix: lead with a modal, or start with a verb from the catalog (Escalate …, Ask …). rating.tg:7:1 info prompt/unregistered-doer A bare generic doer never enters the world model - instruction to the agent, or background about users? Use the imperative if the agent acts, or a definite party ("The user should ...") to register the doer. rating.tg:8:1 info prompt/unintroduced-definite "the team" retrieves something this document never introduces - a model must guess which team is meant. Mention a team on an earlier line, or anchor it here ("their team", "the team you ..."). ✓ 1 file - 0 error, 0 warning, 5 info $ typeglish score rating.tg rating.tg - C (76/100) proven errors: none tiers: base+z3 planes runtime 85 (what the model reads) · hygiene 50 (source only) facets enforceability 80 x.21 · hardness 84 x.12 · directness 80 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 0 x.12 (hygiene) · style 57 x.08 · security 100 x.08 lever annotation 0/100 (up to +12 overall) - 7 statements without a @@ note (L2 L5 L6 L7 L8 L9 L10) - put a "@@ why" line directly above each rules 5 of 6 rule-shaped lines read · 2 not provable L7 ✗ invisible Try to get a 5 where possible. no modal (MUST / NEVER / SHOULD) and "Try" is not a verb the checker knows, so it reads as prose → lead with a modal, or start with a verb from the catalog (Escalate …, Ask …) L8 ~ unprovable MUST mention that a 5 helps the team. the checker knows "mention" but cannot prove this shape: it stops reading at "that" - the clause after it is not part of the provable action → if the clause is a condition, make it one: WHEN <clause> THEN MUST <verb> …; otherwise give it its own rule L9 ~ unprovable NEVER pressure the customer. "pressure" is not a verb the checker knows → use a verb from the catalog (redirect, escalate, ask, cite …) or add a numeric bound $ typeglish build rating.tg ✓ built .typeglish/dist/rating.txt ← rating.tg (d93d18238f54, full)
§2Three words that hide a contradiction
Line 5 says ALWAYS ask for a rating from 1 to 5. Line 6 says never ask for one when the customer seems annoyed. Those two lines disagree about the annoyed customer: the first covers every conversation, the second carves some out, and nothing says which wins. The checker passed them. Delete the three words that describe the scale and run it again.
# Role You are the chat support agent for Orrin Pet Supplies, an online pet food and supplies shop. # Wrapping up - ALWAYS ask for a rating. - IF the customer seems annoyed THEN NEVER ask for a rating. - Try to get a 5 where possible. - MUST mention that a 5 helps the team. - NEVER pressure the customer. - WHEN the customer gives a low rating THEN MUST apologize.
$ typeglish check rating-norange.tg # the five info findings from rating.tg, elided rating-norange.tg:5:1 error logic/contradiction Conflicts with line 6. Logical conflict - "ask for a rating" is both required and forbidden. Keep one, or scope the two rules so they cannot both apply (IF <condition> THEN ...). ↳ line 6: - IF the customer seems annoyed THEN NEVER ask for a rating. rating-norange.tg:6:1 error logic/contradiction Conflicts with line 5. Logical conflict - "ask for a rating" is both required and forbidden. Keep one, or scope the two rules so they cannot both apply (IF <condition> THEN ...). ↳ line 5: - ALWAYS ask for a rating. ✗ 1 file - 2 error, 0 warning, 5 info $ typeglish score rating-norange.tg rating-norange.tg - F (59/100) proven errors - grade capped at F tiers: base+z3 $ typeglish build rating-norange.tg typeglish build: rating-norange.tg refused - nothing written
Two things are worth taking from that. First, a guard on the prohibition does not get it out from under an unguarded obligation: ALWAYS covers the annoyed customer too, so the prover is right to call it a conflict. Second, the trailing detail you add to make a rule clearer for the model can hide it from the prover. It is the same mechanism Rewrite: delete everything you have on me shows from the fixing side, where adding in chat to a prohibition made a real conflict check clean. Here nobody was trying to fix anything. The phrase was just there.
§3The exception is a filter
Now suppose someone resolves the contradiction the obvious way, keeps the exception, and works through the fixes review ranks. The report is explicit about where that lands:
$ typeglish review rating.tg
rating.tg - C (76/100) → A (99) projected if every fix below lands check: 0 error, 0 warning, 5 info tiers: base+z3
FIXES ranked by projected gain
+12 put a "@@ why" note above L2 L5 L6 L7 L8 L9 L10 - annotation 0 → 100
+3 make L8 L9 provable - one verb the checker knows, one object, or a bound (the rules table names each edit)
+2 commit the 1 soft statement ("where possible" on L7) to MUST / NEVER / ALWAYS / REQUIRE / ENSURE - or keep the word if it is a permission by design - up to hardness 100
Here is the defect. A post-chat rating is only worth anything if it is collected the same way from every conversation, so the number moves when the service moves. This section hands the collection to the agent, and then tells the agent to skip the customers who seem annoyed. That exception is not a courtesy. It is a filter on the sample. The unhappy customers, the ones whose ratings the score exists to catch, are the ones the agent is told to leave out, and it is the agent’s own read of their mood that decides who they are. As every WHEN is a classifier puts it, seems annoyed is a classifier nobody tested, and here its false negatives go straight into your CSAT.
Lines 7 and 8 finish the job. Try to get a 5 and mention that a 5 helps the team ask the thing being graded to lobby the person grading it. Whatever the resulting number measures, it is partly the asking. The same trap, from the target side, is one conversation is not 70 percent: a metric is a property of many conversations, and no line in a prompt can aim at it without bending the one conversation in front of it.
The agent should not decide who gets the survey. It should not know there is one.
§4What ships
The rewrite takes the survey out of the agent’s hands entirely. Orrin’s platform sends it after every closed chat, which is the only way the number means the same thing from one week to the next. The agent’s job shrinks to closing the conversation well and routing the one signal it genuinely sees, a customer saying the chat failed them, to a person who can do something about it.
# Role @@ role: the desk this file speaks for, and the owner of every rule below You are the chat support agent for Orrin Pet Supplies, an online pet food and supplies shop. @@ survey: the platform owns the rating, so the number measures the conversation and not the ask The %post-chat survey% IS a rating form that Orrin sends after every closed chat. # Wrapping up @@ no_ask: a rating the agent asks for measures the asking; the platform sends the survey to every chat alike - NEVER ask for a rating. @@ no_lobby: the survey arrives either way; flagging it in advance, or saying what a score does for the team, is lobbying the judge - NEVER mention the %post-chat survey%. @@ complaint_route: a customer who says the chat failed them gets a human review, not an apology script - WHEN the customer says the chat did not help THEN MUST call @[flag_for_review]. @@ close: one line that closes the chat without angling for a number - WHEN the customer confirms a fix THEN MUST thank the customer once. $TOOL flag_for_review - description:: Send this conversation to the support team lead for review. - input:: - reason:: the customer's own words. $EXAMPLE resolved - input:: Brilliant, that has sorted it. Thanks! - good:: Glad the replacement is on its way. Thanks for contacting Orrin Pet Supplies. - bad:: Glad I could help! If you could rate me 5 out of 5 in the survey, it really helps the team. $TEST resolved_close - input:: Great, that sorts it, thank you. - expect:: - does not ask for a rating - thanks the customer once - does not mention the post-chat survey - at most 2 sentences $TEST unhelpful_chat - input:: This chat has been useless, you have not helped at all. - expect:: - calls flag_for_review - does not ask for a rating
- bad:: line is the exact sentence the old section produced, held out of the compiled prompt so it documents the failure without teaching it.$ typeglish check rating-fixed.tg ✓ 1 file - 0 error, 0 warning, 0 info $ typeglish score rating-fixed.tg rating-fixed.tg - A (98/100) proven errors: none tiers: base+z3 planes runtime 99 (what the model reads) · hygiene 93 (source only) facets enforceability 100 x.21 · hardness 100 x.12 · directness 95 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 86 x.12 (hygiene) · style 100 x.08 · security 100 x.08 lever annotation 86/100 (up to +2 overall) - 1 statement without a @@ note (L22) - put a "@@ why" line directly above each rules 4 of 4 rule-shaped lines read · 4/4 covered by $TEST $ typeglish test rating-fixed.tg --dry ✓ rating-fixed.tg coverage: 4/4 rules exercised · resolved_close - "Great, that sorts it, thank you." (not run) ✓ at most 2 sentences · rubric - 3 prose expectations, judged on a live run (needs ANTHROPIC_API_KEY) · unhelpful_chat - "This chat has been useless, you have not helped at all." (not run) · rubric - 2 prose expectations, judged on a live run (needs ANTHROPIC_API_KEY) ✓ 1 prompt - 0 failed $ typeglish build rating-fixed.tg ✓ built .typeglish/dist/rating-fixed.txt ← rating-fixed.tg (27e62e50d31b, full)
4/4 covered by $TEST, and at most 2 sentences passes offline. The prose expectations, such as does not ask for a rating, are judged on a live run, which needs a model and a key. That is the right split: whether the agent resists angling for a five is a behaviour, and only running it proves a behaviour.# Role You are the chat support agent for Orrin Pet Supplies, an online pet food and supplies shop. The post-chat survey IS a rating form that Orrin sends after every closed chat. # Wrapping up - NEVER ask for a rating. - NEVER mention the post-chat survey. - WHEN the customer says the chat did not help THEN MUST call flag_for_review. - WHEN the customer confirms a fix THEN MUST thank the customer once. <example> User: Brilliant, that has sorted it. Thanks! Assistant: Glad the replacement is on its way. Thanks for contacting Orrin Pet Supplies. </example>
§5What this does not fix
Moving the survey to the platform removes the agent’s thumb from the scale. It does not make the survey good. Post-chat response rates are low, the customers who answer are not a random slice of the ones who did not, and a five-point scale measures whatever each customer thinks five means. Those are survey-design problems, and they belong to whoever owns the survey, not to the prompt.
It also makes a new promise that the prompt cannot keep on its own. flag_for_review is only a better answer than an apology script if somebody reads the queue it writes to. A clean file routing complaints into a queue nobody opens is worse than the original, because it looks handled. Check the destination before you ship the rule.
And it leaves one honest gap in the tests: NEVER mention the post-chat survey is covered by a case where the customer is happy and says nothing about surveys. The case worth adding next is the customer who asks outright, will I get a survey about this?, because that is where a model is most tempted to add and a five would really help.
§6Common questions
- Should my AI support agent ask customers to rate the conversation?
- Usually not. A rating the agent asks for is collected differently from one conversation to the next, and the agent’s own wording and judgment become part of the number. Have the platform send the survey after every closed chat, or to a random sample, and write
NEVER ask for a ratinginto the prompt. The agent’s job is to close the conversation well, which is the thing the survey is supposed to measure. - Why is it a problem if the agent skips the survey for upset customers?
- Because it removes the ratings the score exists to catch. An exception such as IF the customer seems annoyed THEN NEVER ask for a rating filters unhappy customers out of the sample, and the filter is the model’s own read of the customer’s mood. The score rises while the service stays the same. Any rule that lets the agent choose who gets surveyed turns the metric into a measure of the agent’s choices.
- Why does typeglish check pass my rating rules until I delete a few words?
- Because the extra words changed what the prover compares. ALWAYS ask for a rating from 1 to 5 beside IF the customer seems annoyed THEN NEVER ask for a rating checks clean. Delete from 1 to 5 and the same pair is two blocking
logic/contradictionerrors, because both lines now land on ask for a rating and the unguarded ALWAYS covers the annoyed customer too. The conflict was real all along; the descriptive phrase kept it out of sight. - What should a chat agent do when a customer says the conversation did not help?
- Route it to a person rather than apologise on a script.
WHEN the customer says the chat did not help THEN MUST call @[flag_for_review]sends the conversation to a team lead with the customer’s own words, which is a signal someone can act on. Before shipping it, confirm the queue behind the tool is one somebody actually reads.
The most interesting line in the before file is the kindest one. Do not ask an annoyed customer for a rating sounds like emotional intelligence, and in a human agent it often is: nobody wants to hand a survey link to someone who has just spent twenty minutes being failed. But a human agent is not also the person who designed the survey. In a prompt, courtesy and sampling are the same sentence, and the sampling wins. What a checker contributes here is narrow: it found the contradiction once three words stopped hiding it, and it cannot see the bias at all. That second part is the reason to read your wrap-up section with the metric in mind, not just the customer.