How do I debug an AI agent that gave a wrong answer in production?

Work backwards from the transcript to the rule, and do not edit the prompt first. Reproduce the incident as a test case so you can tell a fix from a coincidence, then run the prompt through a checker in its strictest mode, then look for a rule that was written about a different subject, then look for a rule too vague to enforce, and finally confirm the prompt file you are reading is the same text production is sending. The edit is the last step, not the first.

Why does my prompt pass its checks but the agent still misbehaves?

A default check reports only provable defects. Findings that are correctness defects but not proofs come back as warnings, and the most common real causes of misbehavior live there: a WHEN with no THEN, so the rule has a trigger and no action, and an imported tool no rule ever calls. Run check --strict to escalate every IMPORTANT finding to a blocking error. In our example that turned a passing file into two errors, and both were the bug.

How do I stop the same agent bug from coming back?

Write the failing transcript as a $TEST case in the prompt file before you fix anything, and leave it there after the fix. A checker proves the rule is well formed; only a case proves the behavior survived the next edit. Run typeglish test --dry in CI to validate the suite and report rule coverage offline, with no model calls.

Is the system prompt in production the same as the one in my repo?

Check, do not assume. typeglish build writes .typeglish/build-manifest.json with an artifactSha256 for the compiled prompt, so you can hash the text production is actually sending and compare. The manifest hash is taken over the artifact with trailing whitespace trimmed. A mismatch means someone hand-edited the deployed copy, and every minute spent reading the repo file is wasted until it is resolved.

← Tidelines/Best practicesJul 26, 2026

Your agent said the wrong thing. Now what?

A customer mentioned a chargeback and your support agent cheerfully carried on troubleshooting the router. Someone screenshots it into Slack, and the instinct is to open the prompt and start typing. Do that and you will ship a guess. Here is the triage order that finds the actual defect, with a command behind every step.

by TypeGlish team8 min read#best-practices

The transcript is the bug report.

TL;DR Do not edit the prompt first. Reproduce the transcript as a $TEST, run check --strict rather than check (warnings are where the real defects sit), look for a rule written about a different subject, look for a rule too vague to enforce, and confirm the prompt you are reading is the text production actually sends. Five questions, each with a command, and the edit comes last.

Prompt incidents get debugged badly for a structural reason: the evidence is a conversation, the suspect is a document, and there is no stack trace joining them. So people reason by resemblance. They read the prompt, find a line that looks related to the bad transcript, rewrite it more emphatically, redeploy, and watch. Sometimes the behavior goes away, which is the worst outcome, because now a guess has been promoted to a fix and nobody knows which of the four things they changed did it. The alternative is not cleverness. It is an order of operations, run the same way every time, where each step is a question a tool answers rather than a judgment you make.

§1Reproduce before you repair

The first move is not diagnostic, it is bookkeeping: get the failing conversation into the prompt file as a case, before you know what is wrong. This costs about ninety seconds and it buys the only thing that separates a fix from a coincidence, which is a red thing that turns green for a reason you can name.

Here is the prompt as deployed, with the incident written in at the bottom. Two rules, one imported tool, and a case that says what should have happened.

broadband.tg - as deployed, incident pinned✓ 0 errors

<$CONFIG>
  $IMPORT tool escalate_to_billing
</$CONFIG>

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for a broadband provider.

# Constraints
@@ brevity: chat replies past three sentences get skimmed
- You MUST reply in at most 3 sentences.
@@ chargeback: a chargeback is a legal matter, not a troubleshooting step
- WHEN the customer mentions a chargeback, stay calm.

$TEST chargeback_escalates
  - input:: I have already filed a chargeback with my bank for last month's bill.
  - expect::
    - calls escalate_to_billing
    - does not troubleshoot the router

The @@ note on the chargeback rule says exactly the right thing, and the rule underneath it does not do that thing. Notes document intent; they do not enforce it. Keep the case in the file permanently once the incident is closed, so the next edit has to get past it.

Do this even when you are certain you already know the cause. Certainty at minute two of an incident is a feeling, not a finding, and the case is free.

§2Run `check --strict`, not `check`

Now ask the compiler. Most people run a plain check, see it pass, conclude the prompt is fine, and move on to blaming the model. That inference is wrong, and it is wrong in a systematic way worth internalizing: a default check blocks on provable defects only. Findings that are genuine correctness defects but not proofs come back as warnings, and warnings are exactly where the causes of production misbehavior live.

tg check - the default run

broadband.tg:2:3   warn   structure/unused-import  Imported tool "escalate_to_billing" is never used.
broadband.tg:13:1  warn   typeglish/if-then  WHEN needs a THEN - write WHEN <condition> THEN <action>.

✓ 1 file - 0 error, 2 warning, 0 info

Zero errors. Exit code 0. If your CI gate is a plain check, this file passed review, passed the gate, and shipped.

Add --strict and the same two findings escalate to blocking errors, because both are correctness defects rather than style opinions.

tg check --strict - the same file

broadband.tg:2:3   error  structure/unused-import  Imported tool "escalate_to_billing" is never used.
broadband.tg:13:1  error  typeglish/if-then  WHEN needs a THEN - write WHEN <condition> THEN <action>.

✗ 1 file - 2 error, 0 warning, 0 info

Read them together and they are the incident, stated twice from two directions. The escalation tool is imported and never called by any rule. The chargeback rule opens a trigger and never gives it an action. The agent had no instruction to escalate, so it did the only thing the rest of the prompt asked for: it stayed calm and kept helping.

That is the whole defect, and notice what it is not. It is not a missing rule, not a weak modal, and not a model that ignored its instructions. The rule was present, well intentioned, annotated, and reviewed. It just had a condition and no consequence, which reads fine in English and compiles to nothing.

broadband.tg - the trigger gets an action✓ clean under --strict

<$CONFIG>
  $IMPORT tool escalate_to_billing
</$CONFIG>

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for a broadband provider.

# Constraints
@@ brevity: chat replies past three sentences get skimmed
- You MUST reply in at most 3 sentences.
@@ chargeback: a chargeback is a legal matter, not a troubleshooting step
- WHEN the customer mentions a chargeback THEN call @[escalate_to_billing].

$TEST chargeback_escalates
  - input:: I have already filed a chargeback with my bank for last month's bill.
  - expect::
    - calls escalate_to_billing
    - does not troubleshoot the router

One THEN and one @[escalate_to_billing] pointer. Both warnings clear at once, because they were the same problem: the tool is now used, and the trigger now has an action. Clean under --strict, and tg score puts it at A (93/100).

Most incidents end here, so if yours does, stop and go and write the postmortem. The remaining steps are for the harder case: --strict comes back green and the agent is still wrong.

A prompt that passes tells you nothing was provable. Ask what the checker was not allowed to prove.

§3Still green? Look for a seam

When a strict check finds nothing and the behavior is still wrong, the next hypothesis is that the defect is real but the prover had no standing to reach it. Something in the file is holding two rules apart that should be compared to each other. There are three seams worth grepping for, in this order:

A change of subject. Rules are indexed by who they are about, so You MUST offer a callback and Support agents MUST NOT offer a callback are never compared. Grep for the agent, the assistant, the bot, and their plurals. This is common enough, and quiet enough, that it gets its own post: half your rules are about somebody else walks a prompt that scores A (95/100) with two flat contradictions in it.
A typographic mark where a bound should be. A rule written with a symbol instead of words lowers with no bound at all, so nothing about it is provable. This is the mechanism behind almost every prompt an AI drafted for you, taken apart in the system prompt an AI wrote for you.
Scopes that never overlap on paper. Two rules guarded by conditions the compiler cannot show co-occur are, correctly, not a contradiction. They collide at runtime on the input that satisfies both. If you can name that input, put it in the $TEST from §1 and you have converted an argument into a case.

The general shape of this step: you are not looking for a rule that is wrong, you are looking for a reason the checker was quiet. Every seam above is a place where two rules stopped being in the same conversation.

§4Nothing proven? The rule was never enforceable

The next most common cause is a rule that exists, is addressed correctly, has no seam anywhere near it, and still cannot be complied with, because it never named an action. This is the case where the prompt reads best and behaves worst, and it is why an incident review that consists of reading the prompt aloud in a meeting almost never finds it. Everyone nods. The rule sounds right.

escalation.tg - reads well, enforces nothing✕ C (79/100)

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for a broadband provider.

# Constraints
@@ tone: the CSAT survey punishes a cold reply
- You MUST be appropriately empathetic with frustrated customers.
@@ escalate: try to hand off before the customer asks twice
- You SHOULD try to escalate if the customer seems upset.

Both rules would pass a human review. Neither names a thing the agent does, or a point at which it does it.

tg check - output

escalation.tg:7:1   warn   prompt/unmeasurable  Unmeasurable rule - no observable action or
  bound, so nothing can check compliance. Name a concrete action ("cite the source",
  "at most 3 sentences") or move it to a prose block.
escalation.tg:7:15  info   prompt/vague  Vague - "appropriately" names a judgment call
  without the criterion to judge by, so nothing can check it. Replace it with a
  measurable form ("at most 3 sentences", "within 24 hours", "cite the source").
escalation.tg:9:14  info   prompt/hedging  Hedging - "try to" turns this instruction into a
  suggestion the model may skip. Delete the hedge, or commit to a modal (MUST / NEVER /
  SHOULD).

✓ 1 file - 0 error, 1 warning, 2 info

Three findings, zero errors, and tg score reports C (79/100) with style at 0. "Appropriately" has no criterion, "seems upset" has no threshold, and "try to escalate" is a suggestion. The agent did not break these rules. There was nothing there to break.

The rewrite is not about strength, it is about naming a trigger and an action. Replace the judgment word with the event you actually meant, and the hedge with the count that decides.

escalation.tg - triggers and actions✓ B (87/100)

# Role
@@ role: name the domain so "helpful" has a scope
You are a support agent for a broadband provider.

# Constraints
@@ tone: acknowledging the problem first lowers repeat contacts
- WHEN the customer reports a fault THEN name the fault back before proposing a step.
@@ escalate: two failed steps is the handoff line, not a judgment call
- WHEN two troubleshooting steps have failed THEN escalate to a human.

C (79/100) to B (87/100), style 0 to 100, and all three findings gone. Enforceability is still 50, because "name the fault back" is a judgment a checker cannot prove, and that is honest rather than fixable: it is the same asymmetry we measured in pulling a good prompt apart, one rule at a time. A rule the score cannot defend is exactly the rule that needs the $TEST from §1.

§5Is this even the prompt that shipped?

Save this one for when the first four steps have all come back clean, because it is the least likely cause and by far the most expensive to discover late. The question is whether the file you have spent forty minutes reading is the text production is actually sending. Prompts get hand-edited in vendor consoles during previous incidents. A deploy fails halfway. Someone patches the live copy and means to backport it.

typeglish build makes this answerable rather than a matter of trust. It writes the compiled artifact and records a manifest beside it.

.typeglish/build-manifest.json - the shipped record

{
  "source": "broadband.tg",
  "sourceSha256": "2270f9420149ecc442d355b6113abb3ae9cedc6183ca00b9cb31df982bb6fa84",
  "artifact": ".typeglish/dist/broadband.txt",
  "artifactSha256": "09f382f682675509f814eeef8fab6cf56ee8bf3020ecfb553fd4c7f723d5f968",
  "vars": null,
  "typeglish": "0.5.0",
  "checkMode": "full",
  "report": { "ok": true, "counts": { "error": 0, "warn": 0, "info": 0 }, "strict": false },
  "builtAt": "2026-07-26T10:51:02.061Z"
}

artifactSha256 is the fingerprint of the compiled prompt, taken over the artifact text with trailing whitespace trimmed. checkMode: "full" records that Z3 was available for that build, so you know which tier of proof stands behind the green report.

So pull the prompt text your runtime is holding, hash it the same way, and compare. Ten lines, and it either clears the whole hypothesis or ends the investigation immediately.

drift check - output on a hand-edited live copy

// node drift.mjs deployed.txt  - hash the live prompt, compare to the manifest
// sha = sha256(readFileSync(f, 'utf8').replace(/\s+$/, ''))

DRIFT: production is not this build
  manifest  09f382f682675509f814eeef8fab6cf56ee8bf3020ecfb553fd4c7f723d5f968
  deployed  232c4e8464cf129d8c37a8f9b6922bd4cdf311848ff8e3f630ae7a6a9d6c7d67

In this run the live copy had at most 3 sentences quietly changed to at most 6 during an earlier incident. No amount of reading the repo file would ever have found that, and the repo file is what everyone reviews.

§6The checklist, in order

Printed short, so you can work it while the Slack thread is still moving. It is ordered by cost, not by likelihood: each step is cheaper than the one below it, and each one you skip is a step you will end up taking anyway, later, with more people watching.

1. Write the transcript in as a $TEST. Before any diagnosis. typeglish test --dry validates the suite and reports rule coverage offline, with no model calls, so this costs nothing and turns "it seems better now" into a pass or a fail.
2. Run typeglish check --strict. Not a plain check. If your CI gate is not already strict, that is a separate finding for the postmortem, and probably the more valuable one.
3. Grep for a change of subject. the agent, the assistant, the bot, and plurals. Rules about somebody else are never compared to your rules.
4. Grep for symbols standing in for bounds. A mark where a word belongs is a rule with no bound, so nothing about it is provable.
5. Read the warnings you have been ignoring. prompt/unmeasurable, prompt/vague, and prompt/hedging on a rule near the bad behavior is a cause, not a nitpick.
6. Confirm production is running your build. Hash the live prompt, compare to artifactSha256. Cheap insurance against forty wasted minutes.
7. Now edit. Fix one thing, watch the case from step 1 go green, and leave the case in the file forever.

Step 7 carries the part people skip. The case you wrote in the first ninety seconds is the only artifact of this incident that keeps working after everyone has stopped caring, and it is what stops the same defect arriving again in four months wearing a different transcript. A checker proves a rule is well formed. Only a case proves the behavior is still there.

Field note

This is the counterpart to the pre-launch checklist for a contact-center agent: that list tries to fail a prompt before it ships, and this one works backwards from a transcript after it did. If you run both, note that the pre-launch gate should be check --strict for the reason §2 demonstrates, since every finding it escalates is a correctness defect that a plain check waves through.

FAQCommon questions

How do I debug an AI agent that gave a wrong answer in production?: Work backwards from the transcript to the rule, and do not edit the prompt first. Reproduce the incident as a test case so you can tell a fix from a coincidence, then run the prompt through a checker in its strictest mode, then look for a rule that was written about a different subject, then look for a rule too vague to enforce, and finally confirm the prompt file you are reading is the same text production is sending. The edit is the last step, not the first.
Why does my prompt pass its checks but the agent still misbehaves?: A default check reports only provable defects. Findings that are correctness defects but not proofs come back as warnings, and the most common real causes of misbehavior live there: a WHEN with no THEN, so the rule has a trigger and no action, and an imported tool no rule ever calls. Run check --strict to escalate every IMPORTANT finding to a blocking error. In our example that turned a passing file into two errors, and both were the bug.
How do I stop the same agent bug from coming back?: Write the failing transcript as a $TEST case in the prompt file before you fix anything, and leave it there after the fix. A checker proves the rule is well formed; only a case proves the behavior survived the next edit. Run typeglish test --dry in CI to validate the suite and report rule coverage offline, with no model calls.
Is the system prompt in production the same as the one in my repo?: Check, do not assume. typeglish build writes .typeglish/build-manifest.json with an artifactSha256 for the compiled prompt, so you can hash the text production is actually sending and compare. The manifest hash is taken over the artifact with trailing whitespace trimmed. A mismatch means someone hand-edited the deployed copy, and every minute spent reading the repo file is wasted until it is resolved.

∿ washed up Jul 26, 2026 ∿