AI agent regression testing is checking every new version of an agent against the failures it has already had, so a failure that was fixed cannot come back unnoticed. The strongest regression tests are built from real production sessions where the agent went wrong, scored by a judge that checks the failure is absent, and compared against a baseline pass rate each time they run.
Why agents need their own kind of regression test
In ordinary software, a regression test pins an input to an expected output. Agents break that model in two ways. Their output varies from run to run, so an exact match fails on answers that are fine. And most of their failures are silent: the run completes without an error, but the answer was wrong, irrelevant or unhelpful. A test suite that only catches crashes misses most of what goes wrong.
Agents also change constantly. A prompt edit, a new model, a changed tool description or a new retrieval source can each bring back a failure someone fixed last month. Every one of those is a change to the agent harness, and every one needs the same question answered before it ships: does the agent still avoid the mistakes it has already made?
Where the test cases come from
The most valuable regression cases come from production, because production shows the failures that actually happen, with the inputs real users actually send. That requires two things before any test exists.
- A live quality signal. New traffic is sampled and scored by judges as it arrives, so failing sessions are marked even when nothing crashed.
- A clear definition of failure. In practice a session counts as failing when it ends in an error, scores below a threshold, or a human reviewer labels it a fail.
Together these produce a failure set: the sessions worth learning from. The rest of the process turns that set into tests.
From a pile of failures to named failure modes
A thousand failing sessions are not a thousand tests. Most are the same few problems repeating. The step that makes regression testing tractable is grouping them.
Failing sessions are embedded by behaviour (what the agent did, where it errored, which checks it failed) and grouped with similar sessions, so a large pile becomes a short list of patterns. A person then promotes a pattern into a named failure mode, such as "refund policy misquoted", with a definition that says what distinguishes it and a severity. Promotion asks the reviewer to look at real example sessions first, so every named failure is grounded in evidence rather than guesswork.
Once failure modes exist, new failing sessions are classified onto them: each one is matched to the single best-fitting mode, or to none if no mode clearly applies. That keeps a running count of how often each named failure happens, which tells you which regression tests matter most.
Anatomy of a failure-derived regression test
A regression test built from a failure mode has four parts, and each one comes from the failure itself rather than from a blank page.
| Part | What it is | Where it comes from |
|---|---|---|
| Dataset | The real sessions where the failure happened | The mode's example sessions, every session classified onto it, and the members of its linked cluster |
| Judge | A rubric that scores whether the failure is absent | Drafted from the mode's definition; a high score means the agent did not fail this way |
| Threshold | The score an item needs to pass | Set on the judge |
| Baseline | The pass rate later runs are compared to | Seeded by the first completed run |
Because the judge scores the failure as absent, the reading is simple: a drop in pass rate means the failure is coming back. The test does not need identical output from run to run. It needs the same failure to stay gone across the whole dataset.
One rule protects the whole approach: a test needs evidence. If a failure mode has no classified sessions, examples or linked cluster yet, there is nothing to build a dataset from, and a test over an empty dataset would pass forever while proving nothing.
Keeping the judge honest
A drafted judge is a starting point, not a verdict. Two habits keep it trustworthy.
First, a person reviews the judge's rubric before it runs. A sensible default is to create the judge disabled, so nothing scores until an expert has read the draft and switched it on.
Second, the judge is measured against people. Domain experts label a sample of sessions pass or fail, and the judge's agreement with them is tracked with precision, recall and Cohen's kappa. A judge that disagrees with the experts is fixed before anyone trusts its pass rates.
Running the tests at release time
A failure regression test earns its keep at three moments.
- After a fix ships. Run the test for that failure to confirm the fix holds against the original failing evidence.
- Before a new agent version ships. Run every active test. A pass rate below baseline shows up as a failing test instead of a silent production incident.
- When a test drops. Look at what changed just before it. Recording each prompt, model and tool change as a deploy event turns a failing test into "pass rate dropped right after prompt v7".
Failure tests pair with a second, wider check. A release evaluation replays a frozen golden dataset against the whole candidate version, scores it with the same judges, and flags items that passed the baseline but fail the candidate. A gate on that result gives CI a pass or fail verdict to block on. Use failure tests for "this specific bug must never return" and release evaluations for "this release, as a whole, is not worse".
Baselines should move only on purpose. When a large fix lands and the new normal is better, re-baseline deliberately so later runs are compared against the improved rate.
After the fix ships
A regression test also changes how fixes are judged. A proposed fix can be proven before merge by running the candidate several times, independently, against the failure's own test and the accumulated regression set, and passing only when every run is green. A person still reviews and merges.
After merge, the question becomes whether the failure actually went down in production. Comparing the failure's real volume in equal windows before and after the deploy, while re-checking that its test still holds, gives each fix a verdict: verified, regressed or inconclusive. And if a failure marked resolved starts appearing again, it reopens rather than staying silently closed.
Common mistakes
| Mistake | What goes wrong | Instead |
|---|---|---|
| Writing cases from imagination | The suite guards failures that never happen | Build each test from real failing sessions |
| Exact-match assertions | Tests fail on good answers that are worded differently | Score with a judge and compare pass rates |
| Trusting an unchecked judge | Pass rates measure the judge's blind spots | Review the rubric and track agreement with experts |
| Tests with no evidence | An empty dataset passes forever | Require real sessions before a test exists |
| Silent re-baselining | A slow decline becomes the new normal | Move a baseline only as a deliberate decision |
Questions
What is a regression test for an AI agent?
A check that a new version of the agent does not bring back a failure it has already had. For an agent it is usually a dataset of real sessions where the failure happened, a judge that scores whether the failure is absent, and a baseline pass rate that every later run is compared against.
Why not write regression tests by hand?
Hand-written cases cover the failures someone imagined. Production shows the failures that actually happen, with real inputs. Building tests from real failing sessions means the suite grows exactly where the agent has already gone wrong.
How do you score a regression test when agent output is not deterministic?
With a judge rather than an exact match. The judge scores whether the specific failure is absent, an item passes when its score meets the judge's threshold, and the test compares the pass rate across the whole dataset against a baseline instead of expecting identical output.
How is a failure regression test different from a release evaluation?
A failure regression test guards one known failure over its own evidence. A release evaluation scores a whole candidate version against a golden dataset and flags items that passed the baseline but fail the candidate. The first says this bug must never return; the second says this release as a whole is not worse.
Can you create a regression test before a failure happens?
Only if there is evidence to build it from. A test needs real sessions, exemplars or a linked cluster behind it; a test over an empty dataset would pass forever and prove nothing.
Sources
- Neens Docs, Eval gates.
- Neens Docs, Issues and failure modes and Failure clustering.
- Neens Docs, Pre-prod evaluations, Eval-verified PR and Fix outcomes.