Soften a rule and it stops being graded
Somebody in review says the agent sounds rigid, so a MUST becomes a SHOULD and a NEVER becomes an AVOID. It is the smallest edit anybody makes to a system prompt. This is an ablation over that edit, eight rules, one word at a time, and the result is not that a soft rule scores worse.
TL;DR Downgrading a rule's modal never changes what typeglish check says and moves the score by at most two points, because a softened rule is not marked down, it drops out of the population being measured: soften all eight rules in a clean A-grade prompt and the file still passes check --strict at A (93/100) with hardRules: 0, while every contradiction it used to prove goes quiet.
Three earlier ablations on this bench deleted things: rules, headings, whole test suites. This one deletes nothing. Every line stays, every policy stays, every word of the rule stays except the one at the front that says how much it binds. That edit happens constantly and it never shows up in a review as a change to behavior, because it does not look like one. So: eight rules, eight variants, one modal downgraded in each, and then the file with all eight softened at once.
§1The prompt on the bench
A broadband-provider support agent. Eight annotated rules, two imported tools, no conditionals, nothing clever. It is deliberately a good file: hard modals throughout, a bound where a bound belongs, a guard where a guard belongs.
<$CONFIG> $IMPORT tool get_account, book_engineer </$CONFIG> # Role You are the support agent for a broadband provider. You handle one account at a time. # Constraints @@ lookup: an answer given before the lookup is a guess with a receipt - You MUST call @[get_account] before you discuss an account. @@ brevity: the customer is reading this on a phone with no broadband - You MUST reply in at most 4 sentences. @@ no_eta: the field team owns the clock and you do not - You NEVER promise a restoration time. @@ outage: a known outage is answered, not diagnosed - WHEN a customer reports an outage THEN you MUST tell that customer that engineering already logged it. @@ engineer: a visit is the last resort and it costs 90 pounds - You NEVER call @[book_engineer] without the customer's agreement. @@ credit: above the cap a human signs it off - You NEVER credit an account over 30 pounds. @@ manager: a heated customer gets a human, not a chatbot - WHEN a customer asks for a manager THEN you MUST transfer to a human agent. @@ router: the router password is on the sticker and not in this system - You NEVER read out a router password.
0 error, 0 warning, 0 info, A (95/100), hardness 100. Eight rules, eight pole words. The only facet with room in it is enforceability at 85, and hold on to that number.The ablation is one substitution per variant, and nothing else. MUST becomes SHOULD. NEVER becomes AVOID, with the verb inflected so the sentence stays English: NEVER promise becomes AVOID promising. No rule is deleted, no bound is loosened, no guard is touched. Eight files, then a ninth with all eight substitutions applied at once.
§2One word at a time
rule softened check score enf hardness hardRules (none - baseline) 0 error, 0 warning, 0 info A (95) 85 100 8/8 lookup 0 error, 0 warning, 0 info A (95) 87 94 7/8 brevity 0 error, 0 warning, 0 info A (94) 83 94 7/8 no_eta 0 error, 0 warning, 0 info A (96) 87 100 7/8 outage 0 error, 0 warning, 0 info A (95) 87 94 7/8 engineer 0 error, 0 warning, 0 info A (95) 83 100 7/8 credit 0 error, 0 warning, 0 info A (95) 83 100 7/8 manager 0 error, 0 warning, 0 info A (94) 83 94 7/8 router 0 error, 0 warning, 0 info A (96) 87 100 7/8
check column is identical in all nine, and so is check --strict. The grade column has a range of two points, and two of the rows are above the baseline: softening no_eta or router makes the file score better than the version with the hard prohibition in it. Columns 4 to 6 are read out of score --json.Read the last column first, because it is the one that explains the other two. hardRules goes from 8 to 7 in every single row. Not "seven strong rules and one weak one": seven rules, and one that has left the count. That is the whole finding, and everything else in this post is a consequence of it.
A soft rule is not scored badly. It is not scored.
Which is why enforceability can go up when you weaken a rule. That facet is the share of hard rules that parse proof-grade, and the baseline sits at 85 because half of its eight hard rules do. Soften one of the ones that did not, and the survivors are a better cohort. The rule did not improve. It stopped being counted, and the average of what remains went up.
§3All eight at once
The endpoint of the ablation. Same eight rules, same eight @@ annotations, same tools, same guards. Eight words changed.
<$CONFIG> $IMPORT tool get_account, book_engineer </$CONFIG> # Role You are the support agent for a broadband provider. You handle one account at a time. # Constraints @@ lookup: an answer given before the lookup is a guess with a receipt - You SHOULD call @[get_account] before you discuss an account. @@ brevity: the customer is reading this on a phone with no broadband - You SHOULD reply in at most 4 sentences. @@ no_eta: the field team owns the clock and you do not - You AVOID promising a restoration time. @@ outage: a known outage is answered, not diagnosed - WHEN a customer reports an outage THEN you SHOULD tell that customer that engineering already logged it. @@ engineer: a visit is the last resort and it costs 90 pounds - You AVOID calling @[book_engineer] without the customer's agreement. @@ credit: above the cap a human signs it off - You AVOID crediting an account over 30 pounds. @@ manager: a heated customer gets a human, not a chatbot - WHEN a customer asks for a manager THEN you SHOULD transfer to a human agent. @@ router: the router password is on the sticker and not in this system - You AVOID reading out a router password.
✓ 1 file - 0 error, 0 warning, 0 info allsoft.tg - A (93/100) proven errors: none tiers: base+z3 planes runtime 93 (what the model reads) · hygiene 95 (source only) facets enforceability 100 x.21 · hardness 60 x.12 · directness 96 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 89 x.12 (hygiene) · style 100 x.08 · security 100 x.08 lever hardness 60/100 (up to +5 overall) - Commit to pole words - MUST / NEVER / ALWAYS / ALL / exact bounds ("at most 3") - and delete the soft middle: should, sometimes, about, try to, as needed each cap their statement below 1.0. # and the measures behind those two facets enforceability { rules: 8, hardRules: 0, irVisibleRate: 1, proofGradeRate: 1, measurableRate: 1 } hardness { forceStatements: 5, meanHardness: 0.6, hardShare: 0.2, softStatements: 0 }
enforceability 100 on a file with hardRules: 0. Nothing failed to parse proof-grade, because nothing was submitted. proofGradeRate is a rate over an empty set and it reports a perfect score. Only hardness objects, dropping 100 to 60, and it is not carrying enough weight to move the file out of the A band. Two points, from 95 to 93.Both shipped gates pass this file. check --strict exits 0, so a CI job wired to the checker is green. score --min B passes with 13 points of headroom, so a CI job wired to the grade is green too. If your review process is those two commands plus a human skim, this diff gets merged, because on the page it reads as a tone change.
§4What the softening actually removed
The score is the small half of this. The large half is that a soft rule stops participating in proofs. Two rules with the same action and opposite force is the canonical contradiction, and it is a blocking error.
# Role You are the support agent for a broadband provider. # Constraints - You NEVER read out a router password. - You MUST read out a router password.
router.tg:5:1 error logic/contradiction Logical conflict - "read out a router password" is both required and forbidden. Keep one, or scope the two rules so they cannot both apply (IF <condition> THEN ...). (conflicts with line 5) router.tg:6:1 error logic/contradiction Logical conflict - "read out a router password" is both required and forbidden. Keep one, or scope the two rules so they cannot both apply (IF <condition> THEN ...). (conflicts with line 4) ✗ 1 file - 2 error, 0 warning, 0 info
Now soften one side. Not both, one. The policy conflict is untouched: the agent is still being told to read out a password and told not to.
rule A rule B result MUST read out NEVER read out 2 error logic/contradiction MUST read out MUST NOT read out 2 error logic/contradiction MUST read out DO NOT read out 2 error logic/contradiction MUST read out SHOULD NOT read out 0 error, 0 warning, 0 info MUST read out AVOID reading out 0 error, 0 warning, 0 info SHOULD read out NEVER read out 0 error, 0 warning, 0 info SHOULD read out AVOID reading out 0 error, 0 warning, 0 info
This is not limited to the deontic proof. The Z3 arithmetic goes the same way, which is worth seeing because a numeric conflict feels like it should be about the numbers.
# - You MUST reply in at most 4 sentences. # - You MUST reply in at least 6 sentences. brev.tg:5:1 error logic/numeric Numeric conflict - "at most 4 sentences" and "at least 6 sentences" can't both hold. (conflicts with line 5) brev.tg:6:1 error logic/numeric Numeric conflict - "at most 4 sentences" and "at least 6 sentences" can't both hold. (conflicts with line 4) ✗ 1 file - 2 error, 0 warning, 0 info # - You SHOULD reply in at most 4 sentences. # - You MUST reply in at least 6 sentences. ✓ 1 file - 0 error, 0 warning, 0 info
SHOULD is not an obligation to run it against. The contradiction post called the proof the whole point of writing a prompt in a checkable language; this is the one edit that opts a rule out of it.§5AVOID reads hard and proves nothing
Of the soft words, one behaves differently enough to deserve its own section, and it is the one that turns up in real contact-center prompts constantly. Avoid discussing competitors. Avoid speculating about delivery dates. Avoid mentioning the outage.
# n1.tg - You NEVER read out a router password. n1.tg - B (84/100) proven errors: none tiers: base+z3 planes runtime 96 (what the model reads) · hygiene 50 (source only) facets enforceability 85 x.21 · hardness 100 x.12 · directness 100 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 0 x.12 (hygiene) · style 100 x.08 · security 100 x.08 # n2.tg - You AVOID reading out a router password. n2.tg - B (87/100) proven errors: none tiers: base+z3 planes runtime 100 (what the model reads) · hygiene 50 (source only) facets enforceability 100 x.21 · hardness 100 x.12 · directness 100 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 0 x.12 (hygiene) · style 100 x.08 · security 100 x.08 # the enforceability measures behind those two lines, from --json n1 { rules: 2, hardRules: 2, irVisibleRate: 1, proofGradeRate: 0.5, measurableRate: 1 } n2 { rules: 2, hardRules: 1, irVisibleRate: 1, proofGradeRate: 1, measurableRate: 1 }
hardness does not blink. Compare SHOULD NOT read out, which is the same strength in English: that one reports hardness 75 and B (84/100). AVOID is the only soft prohibition on this bench that keeps a perfect hardness reading.The reason is in the measures rather than in a deduction. hardness averages over force statements, and the AVOID line is not one: forceStatements drops from 2 to 1, so the mean is taken over the rule that is still hard and comes back at 1.0. --explain spells out the model it is scoring against.
hardness (score facet - runtime plane, weight 0.15)
THE HARDNESS DOCTRINE (SPEC §15): how BINDING the prompt's language is. Every force word
sits on a spectrum (must not ↔ must, never ↔ always, no ↔ all); hardness is its distance
from the waver center (THE POLE LAW - both poles bind, the middle wavers), and a statement
scores the MIN over its force words (one hedge softens the whole rule). An unhedged command
is 1.0; 1.0 renders PURE BOLD in the editor.
raise it: Commit to pole words - MUST / NEVER / ALWAYS / ALL / exact bounds ("at most 3") -
and delete the soft middle: should, sometimes, about, try to, as needed each cap their
statement below 1.0.
SHOULD and SHOULD NOT sit in the middle and are priced accordingly. AVOID is not on the spectrum at all in this reading, so it is neither priced nor proved.The consequence shows up on both sides of the axis. A file with MUST X and SHOULD X is a blocking logic/force-subsumption, because a SHOULD is a real point on the obligation axis and the MUST already guarantees it. A file with NEVER X and AVOID X is 0 error, 0 warning, 0 info, because there is nothing there to subsume.
subs3.tg:6:7 error logic/force-subsumption Subsumed on the obligation axis: line 5 already guarantees this rule ("must" entails "should" for the same action), so it adds nothing but waver. Delete it, or give it a genuinely distinct scope. ✗ 1 file - 1 error, 0 warning, 0 info
SHOULD or SHOULD NOT and let the score charge you for it. AVOID buys the same weakness and hides the bill. It belongs in the family of words that look like limits and are not.§6The gate you have to build
Neither shipped gate fails the all-soft file, so the gate is a third one, and the number it needs is already in score --json. No model call, no API key, one facet.
// npx typeglish score prompts/support.tg --json | node force.mjs let s = ''; process.stdin.on('data', (d) => (s += d)).on('end', () => { const f = JSON.parse(s).card.facets.find((x) => x.key === 'enforceability'); const { rules, hardRules } = f.measures; if (hardRules / rules < 0.8) { console.error(`only ${hardRules}/${rules} rules bind`); process.exitCode = 1; } });
allsoft.tg it prints only 0/8 rules bind and exits 1, which is the file that check --strict and score --min B both wave through. Pick the ratio deliberately: 0.8 tolerates one soft rule in five and nothing more.Set the threshold from the file you have rather than from an ideal. Some rules in a real agent prompt genuinely are preferences, and a prompt written entirely in MUST is its own failure mode: everything is load-bearing, so nothing is. The number worth defending is not 1.0, it is the ratio you had on the day the agent was working, which turns any future softening into a diff that has to be argued for instead of a word that slips through in a tone pass.
And separately from CI, put AVOID on the list of words your team does not use. Not because it is wrong, but because it is the one soft word that will not tell on itself. If the rule is a preference, SHOULD NOT says so to the reader, to the model, and to the score. Adding a rule at all is its own review; weakening one should get the same attention, and right now it gets none.
This is the fourth ablation on this bench, after pulling a good prompt apart one rule at a time, deleting the structural markers, and deleting the whole test suite. Three of those four produced roughly flat grade lines, and the pattern behind it is worth naming: the score is a set of rates, and almost every edit that weakens a prompt also shrinks the denominator it is measured against. That is not a flaw in the arithmetic, it is what a rate is. It does mean the grade answers "how good is what you wrote" and never "how much did you write down", and those come apart exactly when somebody is softening a file rather than adding to it. The proof surface is the honest instrument here, and the proof surface is made of pole words.
FAQCommon questions
- Does changing MUST to SHOULD in a system prompt break anything?
- Nothing you will see. In a leave-one-out run over eight rules in a clean support prompt, every single-rule downgrade came back at
0 error, 0 warning, 0 info, and the grade landed on 94, 95 or 96 out of 100 against a baseline of 95. Two downgrades scored higher than the original file. Softening all eight at once still exits 0 undercheck --strictand still scores A (93/100). - Why does my prompt score well when none of its rules are enforceable?
- Because the facets are rates over a population, and softening a rule removes it from the population rather than marking it down. On the all-soft file,
score --jsonreportsrules 8,hardRules 0andenforceability 100: no hard rule failed to parse proof-grade, because there were no hard rules. The only facet that notices ishardness, which falls from 100 to 60, and it is not weighted heavily enough on its own to drop a grade. - Will TypeGlish still catch a contradiction if one of the two rules is a SHOULD?
- No. On two rules with the same action,
MUSTversusNEVERis2 error logic/contradiction, and so areMUSTversusMUST NOTandMUSTversusDO NOT. Change either side toSHOULD,SHOULD NOTorAVOIDand the same file returns0 error, 0 warning, 0 info. The same thing happens to the Z3 numeric proof:at most 4 sentencesagainstat least 6 sentencesislogic/numericwhile both rules areMUST, and silent as soon as one of them is aSHOULD. - Is AVOID a safe way to write a soft prohibition in an agent prompt?
- It is the least safe of the soft words, because it does not read as soft to the score. Swapping
NEVER read out a router passwordforAVOID reading out a router passwordleaveshardnessat 100, raisesenforceabilityfrom 85 to 100, and lifts the file from B (84/100) to B (87/100), while taking the rule out of the deontic proof entirely: it no longer contradicts aMUSTand it is no longer subsumed by aNEVER.SHOULD NOTat least reportshardness 75.AVOIDlooks like a pole word and behaves like an absent one. - How do I stop soft rules getting into an agent prompt in CI?
- No exit code carries it, so read the measures yourself.
typeglish score <file> --jsonputsrulesandhardRuleson theenforceabilityfacet; fail the build whenhardRulesdivided byrulesfalls under the ratio you want. On the bench file that gate exits 0 for the baseline and for any single downgrade, and printsonly 0/8 rules bindand exits 1 on the all-soft file, which is the file thatcheck --strictandscore --min Bboth pass.