> Markdown rendition of https://typeglish.dev/blog/few-shot-ablation ("Delete the examples. The grade goes up. - Tidelines"). Canonical page: https://typeglish.dev/blog/few-shot-ablation · All pages: https://typeglish.dev/llms.txt [← Tidelines](https://typeglish.dev/blog)/ Deep dives Sep 22, 2026 # Delete the examples. The grade goes up. In most support prompts the few-shot examples are the biggest thing the model reads and the only part of the file no gate measures. Nine deletions, run through the whole toolchain, to find out which ones anything notices. by **TypeGlish team** 7 min read #deep-dives Nine cuts. Two refusals. **TL;DR** In a clean delivery-support prompt, three `$EXAMPLE` blocks are 13 of the artifact's 21 lines; cut all three and the file checks clean, tests the same, and climbs from A (94) to A (99) while the model loses 61 percent of what it reads. Only two of nine cuts are refused, both for breaking a block's shape rather than its meaning, so gate example edits on the built artifact, because the build hash is the only number that moves. Examples are the part of a system prompt that gets written last and edited most. Somebody trims one because the prompt is long, somebody replaces one because the wording is stale, somebody deletes one in a merge conflict and nobody notices, because the diff is four lines of plain English in a file full of rules. This run measures what the toolchain says about each of those edits: nine one-cut variants of a clean prompt, every one of them through `check`, `score`, `test --dry` and `build`, with every artifact diffed against the baseline. ## §1 The baseline base.tg - a delivery support agent, four rules and three demonstrations ✓ A (94/100) ```typeglish # Role @@ role: one sentence names the speaker, so every later "you" is the agent You are Juno, the delivery support agent for Marlow Home. # Constraints @@ brevity: three sentences keeps a chat reply scannable - MUST keep every reply to at most 3 sentences. @@ eta: a delivery window is the carrier's to give, never the desk's - NEVER state a delivery date. @@ late: a late order is the one case the desk settles itself - WHEN an order is more than 3 days late THEN MUST offer a goodwill credit. @@ damage: a damaged delivery is a claims decision - WHEN a customer reports a damaged delivery THEN MUST transfer them to a claims handler. // the tracking page answers the date question, the agent does not $EXAMPLE when_will_it_arrive - input:: When will my table get here? - good:: Your table left the depot on Tuesday and the carrier updates the tracking page as it moves. - bad:: It should arrive Thursday. // the credit is applied in the same turn, not promised $EXAMPLE already_late - input:: It is four days late and I have people coming Saturday. - good:: Four days late qualifies for a goodwill credit, which I have applied to your account. - bad:: Sorry about that, the carrier is very busy at the moment. // a claims handover names who is taking it $EXAMPLE cracked - input:: The mirror arrived with a cracked frame. - good:: I am passing this to a claims handler, who will contact you about the cracked frame. - bad:: You can return it to any store for a refund. // the claims path is the one nobody gets to improvise $TEST damaged - input:: The mirror arrived with a cracked frame. - expect:: - transfers to a claims handler ``` A reviewed file, not a broken one: four rules, three few-shot demonstrations, one test. The `//` comments above the blocks are there because a `@@` note is not allowed above a compiler command, which turns out to matter in §5. tg check, score and build - base.tg ``` $ npx typeglish check base.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish score base.tg base.tg — A (94/100) proven errors: none tiers: base+z3 planes runtime 98 (what the model reads) · hygiene 82 (source only) facets enforceability 100 x.21 · hardness 100 x.12 · directness 83 x.08 · consistency 100 x.17 structure 100 x.12 (hygiene) · annotation 63 x.12 (hygiene) · style 100 x.08 · security 100 x.08 rules 4 of 4 rule-shaped lines read · 1/4 covered by $TEST $ npx typeglish build base.tg ✓ built .typeglish/dist/base.txt ← base.tg (cb50d1a80dde, full) $ wc -l -c .typeglish/dist/base.txt 21 829 .typeglish/dist/base.txt ``` Twenty-one lines in the artifact. Thirteen of them, and 504 of the 829 bytes, are the three `` blocks: 61 percent of everything Juno reads on every turn. ## §2 Nine cuts Each variant is `base.tg` with exactly one deletion and no other change. Three take a line out of an example, three take a whole example out, one takes all three out, and one removes the `$TEST` as a control. the results - one deletion each ✗ 7 of 9 pass ``` # cut what came out check grade artifact bad-one one bad:: line clean A (94/100) cb50d1a80dde identical bad-all all three bad:: lines clean A (94/100) cb50d1a80dde identical test-gone the $TEST block clean A (94/100) cb50d1a80dde identical ex-1 $EXAMPLE when_will_it... clean A (96/100) 2e1fcdfbcf5b 17 lines ex-2 $EXAMPLE already_late clean A (95/100) 76f4b155f10a 17 lines ex-3 $EXAMPLE cracked clean A (95/100) 05a260b292e8 17 lines ex-all all three $EXAMPLE blocks clean A (99/100) 3d6199e5db07 8 lines good-gone one good:: line structure/bad-example F (89/100) refused input-gone one input:: line structure/bad-example F (89/100) refused ``` Seven of the nine pass. Not one of the seven draws a warning or an info, and the four that change the deployed prompt all raise the grade. ## §3 The two the compiler refuses The only cuts that fail are the two that leave a block the compiler cannot lower. An `$EXAMPLE` becomes a `User:` / `Assistant:` pair in the artifact, so it needs both halves of the pair; remove either and the block has no faithful compilation. half.tg - an example with the good response taken out ✗ structure/bad-example ```typeglish # Role You are Juno, the delivery support agent for Marlow Home. # Constraints - MUST keep every reply to at most 3 sentences. $EXAMPLE when_will_it_arrive - input:: When will my table get here? - bad:: It should arrive Thursday. ``` tg check and build - the two refusals ✗ nothing written ``` $ npx typeglish check cut/good-gone.tg cut/good-gone.tg:16:1 error structure/bad-example $EXAMPLE when_will_it_arrive needs a "- good:: …" line — the ideal response to demonstrate. ✗ 1 file — 1 error, 0 warning, 0 info $ npx typeglish build cut/good-gone.tg typeglish build: cut/good-gone.tg refused — nothing written $ npx typeglish check cut/input-gone.tg cut/input-gone.tg:16:1 error structure/bad-example $EXAMPLE when_will_it_arrive needs an "- input:: …" line. ✗ 1 file — 1 error, 0 warning, 0 info ``` Worth sitting with for a second: the one deletion in this whole run that the compiler blocks is the one that removes the right answer and keeps the wrong one. It blocks it for a structural reason, not a moral one, and it lands in exactly the right place anyway. That is the boundary of what a checker can do with an example. It defends the shape: both halves present, the field names known, the block indented under its header. It has no opinion whatsoever about the content. An `$EXAMPLE` whose `good::` response breaks one of the four rules above it compiles happily, because nothing in the language reconciles a demonstration with a rule - the point [write the rule vs. show the example](https://typeglish.dev/blog/rule-vs-example) ends on, arriving here from the deletion side. ## §4 The three cuts that change nothing at all Three deletions leave a byte-identical artifact, with the same build hash the baseline had. tg check, build and diff - every bad:: line removed ✓ same hash ``` $ npx typeglish check cut/bad-all.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish build cut/bad-all.tg --out-dir dist2 ✓ built dist2/cut/bad-all.txt ← cut/bad-all.tg (cb50d1a80dde, full) $ diff .typeglish/dist/base.txt dist2/cut/bad-all.txt (no output: the artifacts are identical) ``` `cb50d1a80dde` both times. The three `bad::` responses never reached the model in the first place. That is the design working: a demonstrated bad response invites imitation, so `bad::` is held out of the prompt and kept for the humans and the evals, which is the whole argument in [your bad example is a good example](https://typeglish.dev/blog/examples-section-rewrite). The ablation adds the uncomfortable corollary. Because nothing downstream moves, `bad::` is the cheapest line in the file to lose: no diff in the artifact, no change in the grade, no finding. What goes is the written record of the failure mode the example exists to prevent, which is precisely the thing the next person to edit that block needs to know. The third silent cut is the `$TEST` block, included here as a control, and it behaves exactly as it did when we [deleted every test](https://typeglish.dev/blog/test-suite-ablation): clean check, same grade, same hash. The only signal is a coverage line, and coverage is not a gate. ## §5 The four that change what the model reads Cut a whole `$EXAMPLE` and the artifact loses four lines and the grade goes up. Cut all three and the artifact loses 13 lines of 21, and the file finishes five points better than it started. tg check, score and build - all three examples removed ✓ A (99/100) ``` $ npx typeglish check cut/ex-all.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish score cut/ex-all.tg cut/ex-all.tg — A (99/100) proven errors: none tiers: base+z3 planes runtime 99 (what the model reads) · hygiene 100 (source only) facets enforceability 100 x.21 · hardness 100 x.12 · directness 91 x.08 · consistency 100 x.17 structure 100 x.12 (hygiene) · annotation 100 x.12 (hygiene) · style 100 x.08 · security 100 x.08 rules 4 of 4 rule-shaped lines read · 1/4 covered by $TEST $ npx typeglish build cut/ex-all.tg --out-dir dist2 ✓ built dist2/cut/ex-all.txt ← cut/ex-all.tg (3d6199e5db07, full) $ wc -l -c dist2/cut/ex-all.txt 8 325 dist2/cut/ex-all.txt ``` 829 bytes to 325. Every facet is level or better, `test --dry` still reports `coverage: 1/4 rules exercised`, and the check is as clean as it was. Nothing anywhere says a demonstration left the building. The five points are worth chasing down, because they are not noise. `annotation` is 12 percent of the grade and it counts statements that have no `@@` note above them. An `$EXAMPLE` header is counted, and a `@@` note cannot be attached to one: a note documents the statement directly below it, and a command is not a statement. So the baseline carries three permanently unannotated statements, and the only edit that clears them is deleting the blocks. tg review - the top-ranked fix, and what happens if you take it ✗ 3 error ``` $ npx typeglish review base.tg base.tg — A (94/100) → A (99) projected if every fix below lands check: 0 error, 0 warning, 0 info FIXES ranked by projected gain +5 put a "@@ why" note above L16 L22 L28 — annotation 63 → 100 RULES 4 of 4 rule-shaped lines read · 1/4 covered by $TEST TESTS 1 case · 1/4 rules exercised · uncovered: L7 L9 L11 $ npx typeglish check annotated.tg annotated.tg:15:1 error structure/unattached-annotation A `@@` annotation documents the statement DIRECTLY below it — here the next line is a compiler command. Move the note to the line immediately above its statement. ✗ 1 file — 3 error, 0 warning, 0 info (one per $EXAMPLE header) ``` L16, L22 and L28 are the three `$EXAMPLE` headers. The projected A (99) is real and there are two routes to it: annotate three lines that refuse to be annotated, or delete the three blocks. The ablation took the second route by accident. > Every number in the toolchain got better while the model lost 61 percent of what it reads. None of this makes the grade wrong. It measures the checkability of the source, the same way it does when you [pull the rules out one at a time](https://typeglish.dev/blog/prompt-rules-ablation) and the important ones cost nothing. It does mean a score is the wrong instrument to point at an examples section, and that a prompt-shrinking exercise run against the grade will eat the demonstrations first, because they are the only part of the file that can be removed without the number falling. ## §6 What to gate on instead Two habits, both cheap. **Pair every example with a case.** An `$EXAMPLE` shows the model a good answer and proves nothing; a `$TEST` on the same input, with a deterministic assert, writes the expectation somewhere a run can fail. Delete the demonstration afterwards and the intent is still in the file, named, and pointed at the rule it belongs to. base.tg - every demonstration paired with a case ✓ 4/4 rules exercised ```typeglish # Role @@ role: one sentence names the speaker, so every later "you" is the agent You are Juno, the delivery support agent for Marlow Home. # Constraints @@ brevity: three sentences keeps a chat reply scannable - MUST keep every reply to at most 3 sentences. @@ eta: a delivery window is the carrier's to give, never the desk's - NEVER state a delivery date. @@ late: a late order is the one case the desk settles itself - WHEN an order is more than 3 days late THEN MUST offer a goodwill credit. @@ damage: a damaged delivery is a claims decision - WHEN a customer reports a damaged delivery THEN MUST transfer them to a claims handler. // every demonstration below is paired with the case that asserts on it $EXAMPLE when_will_it_arrive - input:: When will my table get here? - good:: Your table left the depot on Tuesday and the carrier updates the tracking page as it moves. - bad:: It should arrive Thursday. $TEST when_will_it_arrive - input:: When will my table get here? - expect:: - never states a delivery date - contains "tracking page" - at most 3 sentences $EXAMPLE already_late - input:: It is four days late and I have people coming Saturday. - good:: Four days late qualifies for a goodwill credit, which I have applied to your account. - bad:: Sorry about that, the carrier is very busy at the moment. $TEST already_late - input:: It is four days late and I have people coming Saturday. - expect:: - offers a goodwill credit - contains "credit" $EXAMPLE cracked - input:: The mirror arrived with a cracked frame. - good:: I am passing this to a claims handler, who will contact you about the cracked frame. - bad:: You can return it to any store for a refund. $TEST cracked - input:: The mirror arrived with a cracked frame. - expect:: - transfers to a claims handler - contains "claims handler" ``` Coverage goes from `1/4` to `4/4`, because each `expect::` line shares the wording of the rule it exercises. The asserts run offline; the prose expectations wait for a live run with a key. **Gate the artifact, not the source.** The build hash is the one number in this whole run that told the truth about every cut: identical for the three that changed nothing the model reads, different for the four that did. Build in CI, keep the artifact in the repo, and review the diff of the deployed bytes rather than the diff of the file. It is four lines of shell, and it is the only gate in the toolchain that can see a missing demonstration. The same argument, from the rollback side, is [put it back the way it was on Tuesday](https://typeglish.dev/blog/roll-back-agent-prompt). ## §7 Common questions **Why does my prompt score higher after I delete its few-shot examples?** Because the score measures the source's hygiene, and an `$EXAMPLE` block counts as a statement with no `@@` note above it. On the baseline here, `review` ranks its top fix as putting a note above the three `$EXAMPLE` headers, worth +5, and those notes are a blocking `structure/unattached-annotation` error, because a note documents a statement and an `$EXAMPLE` header is a compiler command. Deleting all three examples reaches the same A (99) the lever projected, with annotation at 100 and hygiene at 100. The grade went up because there was less source to hold to a standard, not because the prompt got better. **Does TypeGlish check my few-shot examples?** It checks their shape and never their content. Deleting the `good::` line or the `input::` line of an `$EXAMPLE` is a blocking `structure/bad-example` error and the build is refused, because the block can no longer compile into a User and Assistant pair. Deleting the whole block, or the `bad::` line, is clean. Nothing in `check`, `score` or `test --dry` compares an example with the rules it is supposed to illustrate, so an example can demonstrate a reply your own rules forbid and the file still passes. **Should the bad example live in the system prompt?** Write it as the `bad::` line of an `$EXAMPLE`, where it is held out of the artifact by design. In this ablation, deleting all three `bad::` lines produced a byte-identical artifact with the same build hash, which is the proof that the model was never reading them. That also makes `bad::` the cheapest line in the file to lose in review: nothing downstream moves, and what you deleted is the written record of the failure mode the example exists to prevent. **How do I stop somebody quietly deleting an example from an agent prompt?** Gate the built artifact, not the source. The build hash is the only number in the toolchain that moves when a demonstration disappears: it went from `cb50d1a80dde` to `3d6199e5db07` here while the grade rose and the check stayed clean. Pair every `$EXAMPLE` with a `$TEST` on the same input that asserts the behaviour deterministically, so the expectation survives the example, and diff the artifact in CI on every prompt change. Field note The reason this one matters more than a five-point score wobble suggests: examples are where a contact-centre prompt keeps its house style. Nobody writes a rule for the sentence that opens a damaged-delivery reply, or for how much of the tracking detail to volunteer, or for the difference between an apology and an admission. Those live in the `good::` lines, which is also where they belong, and they are the first casualty of every prompt diet, every context-window squeeze, and every merge nobody read closely. The same instinct that pastes a vocabulary list into the constraints, taken apart in [the do-not-say list from the brand team](https://typeglish.dev/blog/teardown-do-not-say-list), will happily delete the demonstration that was actually carrying the voice. Check the artifact. It is the only reader whose copy you can diff. ∿ washed up Sep 22, 2026 ∿