An eval gate is a regression test built from one of your agent's own confirmed failures and used as a release guard. It pairs a dataset of the real sessions where the failure happened with a judge that scores the failure as absent, and tracks a baseline pass rate that every later run is compared against. When the pass rate falls below the baseline, the failure is coming back.
Why agents need release gates
An agent changes often. A prompt edit, a model upgrade, a new tool description or a routing tweak can each fix one problem and quietly bring back another. Conventional tests do not catch this well: the agent's output varies from run to run, and the failures that matter are rarely the ones someone thought to write a test for.
The failures that matter are already on record. They are in the traces of conversations that went wrong in production. A release gate that is built from those conversations tests exactly what broke before, which is the thing most likely to break again. It is also the step that turns a fixed failure into a permanent check in your agent harness, rather than a fix that holds until the next change.
What an eval gate is made of
In Neens, a gate binds four things together, each with a clear job.
| Part | What it is | Why it matters |
|---|---|---|
| Failure mode | A named, defined pattern of failure with real evidence sessions behind it | Says what the gate protects against, and why it exists |
| Dataset | The failure's evidence: its exemplar sessions, every session classified into the mode, and the members of any linked failure cluster | The gate tests real conversations, not invented ones |
| Judge | An LLM judge whose rubric scores the failure as absent, so a high score means the agent did not show it | Turns each replayed session into a pass or a fail |
| Deployment | The judge's deployment, created disabled until an expert reviews the draft and enables it | Nothing enforces a verdict a person has not checked |
On top of these, the gate carries a baseline pass rate, seeded by its first completed run, and the pass rate of its most recent run. An item passes when its score meets the judge's threshold, which is 0.7 when the judge does not set one.
Building a gate from a failure
A gate starts from a failure you have already confirmed, not from a blank page.
- Confirm a failure mode. The mode needs evidence behind it: classified sessions, a linked cluster or exemplars. Without evidence, Neens refuses to create the gate, because a gate bound to an empty dataset would pass forever.
- Generate the eval. One action on the mode's issue card builds the dataset from its evidence, drafts the judge, parks a disabled deployment for that judge, and registers the gate with an empty baseline.
- Review and enable the judge. The drafted judge is a starting point. An expert reviews its rubric and enables its deployment when satisfied. Until then, the gate shows the deployment as disabled.
- Run it. The first completed run seeds the baseline pass rate. Every later run updates the latest pass rate and is compared against that baseline.
Reading a gate
Because the judge scores the failure as absent, the direction of the signal is simple: a drop in pass rate means the failure is coming back.
Each gate's detail view explains it on one screen. It shows what the gate checks (the source failure mode with its definition and severity, the judge, the dataset and its size, and whether the judge's deployment is enabled yet), the baseline pass rate beside the latest one with a delta, and the ten most recent runs with who or what triggered each.
Two controls keep gates honest over time. You can pause a gate you are not ready to enforce; it keeps its history and can be activated again. And you can re-baseline a gate when you have changed the bar on purpose, for example after a large fix lands and future runs should be compared to the new normal.
Eval gates and pre-production evaluations
Both guard releases, at different sizes.
| Eval gate | Pre-production evaluation | |
|---|---|---|
| Guards | One known failure | A whole candidate version of the agent |
| Runs on | That failure's own evidence dataset | A frozen golden dataset of prompts |
| Compared with | Its own baseline pass rate | A previous run, or a window of live production quality |
| Question it answers | This specific bug must never return | This release, as a whole, is not worse |
A pre-production evaluation replays the golden prompts against the candidate, scores the results with the same judges you run in production, and counts as regressions the items that passed the baseline but fail the candidate. Its gate takes thresholds, such as a maximum number of regressions and a minimum pass rate, and turns them into a pass or fail verdict your CI can block on. When you run the agent yourself, a CI step starts the run, waits for the verdict and fails the build when the candidate regresses.
Both are powered by the same judges, so a judge you refine for one improves the other. Stress tests connect them from the other side: they generate synthetic scenarios grounded in the failure modes your agent already had, and hand them to the pre-production gate so the next candidate is tested against last quarter's failures before it ships.
Where gates sit in the release lifecycle
A gate is not a one-time check. It appears at every stage between a failure and a fix that holds.
- Before merge. An eval-verified fix is checked against the failure's own evals and an accumulated regression set, and has to pass several independent runs before a pull request opens. A fix that repairs the target failure but breaks a past correction fails. A person always merges.
- At release. Run your active gates, and a pre-production evaluation of the candidate, when a new agent version is about to ship. A drop below baseline shows up as a failing gate instead of a silent production incident.
- After deploy. When a fix pull request opened by Neens is merged, Neens watches the fixed failure in real production traffic for seven days, re-checks that the gate still holds, and marks the fix verified, regressed or inconclusive.
- When a gate drops. Recorded deploy events tie the drop to the prompt, model or tool change that landed just before it, so a failing gate reads as "this deploy did it" rather than a mystery.
Over time the accumulated regression set grows: when a person corrects a judge during review, the corrected example is captured into it. Every correction makes the gate stricter, and each new fix has to hold the whole history, not only its own narrow test.
Questions
How is an eval gate different from a unit test?
A unit test checks a fixed input against a fixed expected output. An eval gate replays the real sessions where a failure happened, scores them with a judge, and compares the pass rate to a baseline, because an agent's output varies from run to run and a single exact match is the wrong test.
How is an eval gate different from a pre-production evaluation?
An eval gate guards one known failure, over that failure's own evidence, with its own baseline. A pre-production evaluation scores a whole candidate version against a golden dataset and reports regressions against a baseline run or a production window. Use the first for this specific bug must never return, and the second for this release, as a whole, is not worse.
Why does the drafted judge start disabled?
Because the draft is a starting point. Nothing runs until an expert has reviewed the judge's rubric and enabled it, so a gate never enforces a verdict no person has checked.
What does a drop in a gate's pass rate mean?
The judge scores the failure as absent, so a lower pass rate means the failure is coming back. The next step is to look at what changed just before the drop: recorded deploy events tie the drop to the prompt, model or tool change that landed before it.
Can a gate block a release automatically?
A pre-production evaluation can: its verdict is a pass or fail your CI can block on, and the command that drives it exits with the gate result. An eval gate runs on demand or as part of your release checks, and a drop below its baseline shows up as a failing gate.
Sources
- Neens docs: Eval gates, Pre-prod evaluations, Stress tests.
- Neens docs: Eval-verified PR, Fix outcomes, What changed.