Two arms, one hypothesis, three artifacts
Eight checks for the week growth asks whether you can A/B test the agent. The experiment is not the traffic split. The experiment is the claim that your two prompts differ in exactly one place, and that is the part a compiler can settle before anybody sees either of them.
TL;DR An experiment arm is a file that $IMPORTs the control and carries the rule under test at its own setting, because an arm can add and can never subtract. Three commands gate it: build the empty arm and require the control's exact artifact hash, check the directory rather than the files so the program pass sees the arm against the control, and diff the two artifacts and require the diff to be your hypothesis. The score is not the referee, and the arm that checks clean on its own is the one to be suspicious of.
The request arrives with a number attached, which is what makes it different from most requests. Containment on cancellations is 41 percent, growth thinks the up-front membership-number ask is what loses people, and they would like to run it as a proper test: half the traffic asks for the number the way it always has, half asks only when the account actually changes, two weeks, and then a decision. Nothing about that is unreasonable. The part nobody has said out loud is that you are about to run two system prompts in production and later attribute a business number to the difference between them, which means the difference between them has to be a thing you can state, and not a thing you assembled by copying a file and editing it.
Eight checks, run against a small gym-membership support prompt. As with most of these lists, the prompt is not broken to begin with. What is missing is the harness.
# Role You are a support agent for Halewood Fitness, a gym membership company. # Constraints @@ credit: the goodwill ceiling finance signed off in March - You MUST apply a goodwill credit of at most 20 pounds. @@ cancel_route: a cancellation always goes to a retention specialist - WHEN a member asks to cancel THEN transfer them to a retention specialist. @@ id_check: a membership number unlocks account context - You MUST ask for a membership number. @@ brevity: three sentences keeps a chat reply scannable - You MUST keep every reply to at most 3 sentences.
§1Before you split the traffic
Check 1: baseline the control and write the hash down. Not the score, though take that too. The hash. It is the only identifier in this pipeline that survives a copy-paste, and everything after this check is measured against it.
$ npx typeglish check base.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish score base.tg base.tg — A (94/100) proven errors: none tiers: base+z3 planes runtime 96 (what the model reads) · hygiene 90 (source only) facets enforceability 88 x.21 · hardness 100 x.12 · directness 94 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 80 x.12 (hygiene) · style 100 x.08 · security 100 x.08 $ npx typeglish build base.tg ✓ built .typeglish/dist/base.txt ← base.tg (bc96712e2fd6, full)
bc96712e2fd6 is the control. Write it in the experiment ticket beside the containment number, because in three weeks somebody will ask what arm A actually was and the answer needs to be twelve characters rather than a git branch name.Check 2: make the arm a file that imports the control, not a copy of it. The reflex is to duplicate the prompt and edit the copy, and it works for about four days. Two files with no relationship drift the moment anybody fixes a typo in one of them, and the drift is invisible because both files still pass. A control and an arm have a relationship, so write it down: the arm declares the control as an import and transcludes its sections.
Check 3: build the empty arm and require the control's exact hash. This is the null test and it is the single highest-value four seconds in the list. Before the hypothesis goes in, the arm should be a harness that adds nothing at all, and "adds nothing at all" has an exact meaning: the same bytes reach the model. Here is what the obvious version of that harness actually does.
<$CONFIG> $IMPORT file "base.tg" as control </$CONFIG> @<control.Role> @<control.Constraints>
$ npx typeglish check arm_b.tg ✓ 1 file — 0 error, 0 warning, 0 info $ npx typeglish build arm_b.tg ✓ built .typeglish/dist/arm_b.txt ← arm_b.tg (48b7d72a5dc4, full) $ diff .typeglish/dist/base.txt .typeglish/dist/arm_b.txt 1d0 < # Role 3,4d1 < < # Constraints
48b7d72a5dc4 against the control's bc96712e2fd6. Headings are not decoration to this compiler and they are not decoration to the model either, which is the whole of your headings decide which rules get checked. Had you put the hypothesis in first, this is a confound you would have shipped and never seen.<$CONFIG> $IMPORT file "base.tg" as control </$CONFIG> # Role @<control.Role> # Constraints @<control.Constraints>
build returns bc96712e2fd6: byte for byte the control. That equality is the claim the experiment rests on, and it is now a command rather than a belief.§2The arm that cannot subtract
Check 4: check the directory, not the file. Now put the hypothesis in. Growth wants arm B to stop asking for the membership number up front, so the arm says so, in the only way an arm can say anything, which is by adding a line.
# appended to arm_b.tg: "- You MUST NOT ask for a membership number." $ npx typeglish check arm_b.tg ✓ 1 file — 0 error, 0 warning, 0 info # exit 0 $ npx typeglish check . arm_b.tg:11:1 error logic/contradiction Conflicts with base.tg line 6. Logical conflict — "ask for a membership number" is both required and forbidden. Keep one, or scope the two rules so they cannot both apply (IF <condition> THEN ...). base.tg:10:1 error logic/contradiction Conflicts with arm_b.tg line 8. Logical conflict — "ask for a membership number" is both required and forbidden. Keep one, or scope the two rules so they cannot both apply (IF <condition> THEN ...). ✗ 2 files — 2 error, 0 warning, 0 info program: 1 root — arm_b.tg (2 files) # exit 1
$IMPORT; a directory target is checked as a program, so the arm meets the control and the finding lands on both true lines. The same machinery is what catches a bad merge in both prompts were clean, the merge is not. Gate the experiment on check . and treat the single-file tick as decoration.Check 5: if the hypothesis is a removal, the rule cannot stay in the control. That error is not an obstacle, it is the design telling you where the boundary goes. An arm composes onto the control, so it can only ever add. There is no syntax for "and ignore rule three", and you would not want one: a prompt whose last paragraph repeals its middle is the defect in the sentence that repeals the rest of your prompt, and it resolves at runtime, differently, one conversation at a time.
So the rule under test leaves the control and appears once in each arm, at that arm's setting. The control keeps everything both arms agree on, which is the honest definition of a control anyway.
<$CONFIG> $IMPORT file "base.tg" as shared </$CONFIG> # Role @<shared.Role> # Constraints @<shared.Constraints> @@ id_check: arm A, the rule under test - the desk has always asked up front - You MUST ask for a membership number. $TEST asks_up_front - input:: I want to cancel my membership. - expect:: - contains "membership number" - at most 3 sentences
- WHEN a member changes an account THEN ask for a membership number. and its own $TEST. Three files, and the only place the word membership number appears as a rule is inside an arm.$ npx typeglish check . ✓ 3 files — 0 error, 0 warning, 0 info program: 2 roots — arm_a.tg (2 files), arm_b.tg (2 files) $ npx typeglish build . ✓ built .typeglish/dist/arm_a.txt ← arm_a.tg (b55345df5cec, full) ✓ built .typeglish/dist/arm_b.txt ← arm_b.tg (5d9c30dc9b19, full) ✓ built .typeglish/dist/base.txt ← base.tg (07336c0b2212, full) ✓ built 3/3
program: 2 roots is the line that says the layout is right: two deployable prompts, each composed from two files. And then the thing to be careful about, which is the title of this post. Three artifacts came out. 07336c0b2212 is the control file built on its own, 280 bytes, and it contains no rule about membership numbers at all, because that rule now lives in the arms. It is a perfectly valid build of a prompt nobody should ever deploy. Name your arms in the deploy config explicitly; do not glob the dist directory.§3One diff, one hypothesis
Check 6: diff the two artifacts and require the diff to be the hypothesis. This is the experiment, written down. Everything before it existed to make this diff trustworthy.
$ diff .typeglish/dist/arm_a.txt .typeglish/dist/arm_b.txt 8c8 < - You MUST ask for a membership number. --- > - WHEN a member changes an account THEN ask for a membership number.
Check 7: a two-hunk diff is two experiments sharing one result. Confounding does not arrive as a decision. It arrives as a second ticket landing in the same week, and somebody reasonably observing that while we are testing the ID ask we may as well try offering the freeze. Both changes are good ideas. Together they are not an experiment.
$ npx typeglish check . ✓ 4 files — 0 error, 0 warning, 0 info program: 3 roots — arm_a.tg (2 files), arm_b.tg (2 files), arm_c.tg (2 files) $ diff .typeglish/dist/arm_a.txt .typeglish/dist/arm_c.txt 8c8,9 < - You MUST ask for a membership number. --- > - WHEN a member changes an account THEN ask for a membership number. > - WHEN a member asks to cancel THEN offer a membership freeze.
8c8 is an experiment and 8c8,9 is a conversation. Make the diff a required attachment on the ticket and the conversation happens before the traffic split rather than after the readout.Check 8: log the artifact hash with the conversation, not the arm name. The analysis at the end of this is a join between transcripts and a variant label, and the label is the weakest link in the chain: it is assigned by the router, it survives renames, and it does not change when somebody edits arm B on the Thursday of week one. The hash does. Emit artifactSha256 from the build manifest alongside whatever variant id your experiment platform uses, and the readout can prove that every conversation in bucket B read the same prompt.
arm_a.tg b55345df5cec sourceSha256 a97e697b5bc3 checkMode full arm_b.tg 5d9c30dc9b19 sourceSha256 cb70e041bce6 checkMode full base.tg 07336c0b2212 sourceSha256 e96b51d8350a checkMode full
sourceSha256 changes when anybody touches the file, including a comment; artifactSha256 changes only when the prompt the model reads changes. The one to log with a conversation is the artifact. The one to watch during a running experiment is both, because a source edit that leaves the artifact alone is fine and a source edit that moves the artifact mid-flight has invalidated your week. That distinction is the subject of the prompt is frozen, the artifact is not.§4What the checker will not tell you
It will not tell you which arm won. It is worth being explicit about that, because the score is sitting right there looking like a verdict, and it is not one. Run both arms through it.
$ npx typeglish score arm_a.tg arm_a.tg — A (92/100) proven errors: none tiers: base+z3 facets enforceability 100 x.21 · hardness 100 x.12 · directness 100 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 33 x.12 (hygiene) · style 100 x.08 · security 100 x.08 $ npx typeglish score arm_b.tg arm_b.tg — B (80/100) proven errors: none tiers: base+z3 facets enforceability 50 x.21 · hardness 100 x.12 · directness 90 x.08 · consistency 100 x.17 · structure 100 x.12 (hygiene) · annotation 33 x.12 (hygiene) · style 100 x.08 · security 100 x.08
enforceability halves because arm B's rule is conditional, which is a fact about the grammar of the sentence. Run the identical harness on a bound instead of a scope, arm A at at most 3 sentences against arm B at at most 6, and both arms come back A (92/100) with identical facets. The score is blind to one kind of change and loud about another, and neither reading is evidence. A score --min A gate in CI would have refused to ship arm B.$ npx typeglish test arm_a.tg arm_b.tg --dry ✓ arm_a.tg coverage: 1/1 rules exercised · asks_up_front — "I want to cancel my membership." (not run) ✓ contains "membership number" ✓ at most 3 sentences ✓ arm_b.tg coverage: 0/0 rules exercised · asks_only_on_change — "I want to cancel my membership." (not run) ✓ matches /^(?!.*membership number)/ ✓ at most 3 sentences ✓ 2 prompts — 0 failed
1/1 against 0/0. The two arms are not being measured against the same denominator, because arm B's rule is a conditional rather than a directive and coverage attributes to rules. Both suites are worth having: each pins the behaviour its arm claims, and the negative lookahead on arm B is the assertion that the ask genuinely stopped happening. Neither number is comparable across the arms, in the way deleting every test found the denominator is never quite your rule count.What you are left with is a small, boring, checkable claim: two prompts, identical except for one line, each with a hash you can log. That claim is not the interesting part of an experiment, which is exactly why it is the part that gets skipped, and why a two-week readout so often ends with somebody saying they think arm B also had the shorter greeting in it.
§5Common questions
- How do I A/B test an AI agent system prompt?
- Put everything that is not under test in one control file, give each arm its own file that imports the control and carries the rule under test at its own setting, and let the compiler produce one deployable artifact per arm. Then run three commands before any traffic moves. Build the arm with nothing in it yet and require the artifact hash to equal the control's, which is the null test that proves your harness adds nothing. Check the directory rather than the files, because a program pass composes the imports and an arm that checks clean on its own can be two blocking errors against the control. Diff the two artifacts and require the diff to be your hypothesis and nothing else. A one-line diff is an experiment; a two-line diff is two experiments sharing one result.
- Should each prompt variant be its own file, or one file with a switch?
- Its own file, importing a shared control. The one-file version puts each arm inside a
$SWITCHarm, and a conflict that exists only inside one arm is invisible to the template pass because the compiler has not chosen an arm yet, so you have to resolve per bag to see it. With one file per arm the conflict is a cross-file finding on the ordinary check, at its true file and line, before anything is built. An arm can only add, though, which is the constraint that decides the layout: if your hypothesis is that a rule should be removed or narrowed, that rule cannot stay in the control. Appending you MUST NOT ask for a membership number to an arm whose control requires it is 2 blockinglogic/contradictionerrors on the directory pass, and the fix is to move the rule out of the control into both arms. - Why does my experiment arm pass typeglish check but fail on the directory?
- Because a single-file check does not compose the
$IMPORT. Runningcheckon one arm file reports0 error, 0 warning, 0 infoand exits 0; runningcheckon the directory expands to the.tgfiles, joins them where they import each other, and reports cross-file findings on their true file and line. An arm holding at most 6 sentences against a control holding at most 3 is a clean file on its own and 2 blockinglogic/quantifier-strengtherrors as a program, with the summary line reading2 files, 2 error, 0 warning, 0 infoandprogram: 1 root. Gate the experiment on the directory pass. The single-file green tick is the one number in this pipeline that means nothing. - Can I use the TG score to pick the winning prompt variant?
- No, and it will mislead you in both directions. Two arms differing only in a bound, at most 3 sentences against at most 6, both come back
A (92/100)with identical facets, so the score cannot see the change at all. Two arms differing in scope, an unconditional rule against the same rule behind aWHEN, come backA (92/100)andB (80/100), becauseenforceabilitydrops from 100 to 50 on a conditional and that is a property of the grammar rather than of containment rate. A score floor in CI would refuse to ship half your experiments for reasons unrelated to the hypothesis. The coverage denominator moves too: the same test suite reportscoverage 1/1 rules exercisedon one arm andcoverage 0/0on the other. Let the checker referee the design and let the experiment referee the outcome.
The reason this layout is worth the twenty minutes is not tidiness, it is what happens at the end. Prompt experiments almost never fail because the statistics were wrong. They fail because six weeks later nobody can reconstruct what arm B was, and the winning variant gets merged into the live prompt by hand, from memory, with the freeze rule that was never part of the test. Every check in this list is really one check wearing different clothes: make the difference between the two prompts a thing the machine holds, rather than a thing a person remembers. The hash holds it, the diff states it, and the program pass refuses the arm that quietly repeals a rule the control depends on. What the compiler cannot hold is the decision itself, and it should not pretend to: 0 error on four files is compatible with an arm that answers two questions at once, and a letter grade is a property of how the rules are phrased rather than of whether they work. If you are about to build this, the prompt is frozen, the artifact is not covers what actually counts as a change while the test is running, and the paragraph that lives in three prompts is the same $IMPORT mechanics used for the thing they were designed for.