Agent observability records what an agent did. Agent evaluation judges whether what it did was good. Remediation changes the agent so the same failure stops happening, and proves the change held. They answer three different questions, and an agent is only as reliable as the weakest connection between them.
Three layers, three questions
The terms get used interchangeably, which hides the fact that each layer produces something the next one needs. The table sets them side by side.
| Layer | Question it answers | Needs | Produces | How far most tooling goes |
|---|---|---|---|---|
| Observability | What did the agent do? | Traces from the agent | Runs, steps, tool calls, errors, latency, cost | Standard |
| Evaluation | Was the result good? | Traces, plus a definition of good | Scores with a pass or fail and a reason | Common, sometimes with a dataset built from a failure |
| Remediation | Will this failure happen again? | Named failure modes with evidence | A proven fix, a record that it held, a regression test | Rare: a person writes the fix and ships it on faith |
Instrument, trace, score: most tools do those three. The expensive part of reliability starts after the score, when someone has to decide what to change and whether the change worked.
Observability: what happened
Observability is the recording layer. A trace is one end-to-end run of the agent: a request comes in, the agent works, a response goes out. Each trace is made of spans, one per step: a model call, a tool call, a retrieval, a guardrail check. Each span carries its inputs, outputs, timing and status, and model calls carry the model name and token usage, which is what turns into cost. Traces that share a conversation id roll up into a session, so a multi-turn conversation reads as one unit.
Most agents emit this data through OpenTelemetry or OpenInference, often with configuration rather than code. Once it lands, you can open any run and see exactly which tool was called with which arguments and what came back.
Observability also catches the failures that are visible in the data itself: error rates that jump, latency that doubles. Those are real signals. What observability cannot tell you is whether a run that completed without an error gave the user a correct answer. A refund issued for the wrong amount looks perfectly healthy in a trace.
Source: LangChain, State of Agent Engineering, n=1,340, December 2025. Figures are among organizations with agents in production.
Evaluation: was it good
Evaluation adds a verdict. A judge is an automated evaluator that reads a trace and scores it against a metric, such as faithfulness, relevance, or a rule specific to your business. The output is a score: a number between 0 and 1, an optional pass or fail against a threshold, and a rationale explaining the verdict. Judges are usually model-based and can be combined into composites.
In production, evaluation runs in three ways. Continuously, on a sample of incoming traffic, so there is a quality trend from the first day. On a schedule. And on demand, when you are investigating a specific question. A drop in the pass rate between two windows is itself a signal worth alerting on.
Two things decide whether evaluation is worth trusting. The first is whether the judge agrees with people. A judge is only useful if it reaches the same verdict a domain expert would, so the expert labels a sample and the agreement rate between judge and human is measured and tracked. The second is what happens to a failing score. One failing run is an anecdote. A thousand are a pile. Neither is something you can fix.
That is why diagnosis sits between evaluation and remediation. Failure clustering groups similar failing sessions together, so a thousand bad traces become a short list of patterns. Each pattern becomes a named failure mode, with the real sessions behind it as evidence. The failure mode, not the individual score, is the thing remediation works on.
Remediation: will it happen again
Remediation is the layer that changes the agent. A remediation is a typed, tracked fix for a failure mode, such as a prompt change or a tool fix, generated from the trace evidence rather than from a guess. Done properly it has five parts.
- Find where the failure lives. Not every failure is the agent's fault. The evidence can show that the agent reasoned badly, that a tool contract is wrong, that a downstream service returned an error, or that a guardrail fired exactly as intended. Only the first two call for a change in your repository. A downstream outage becomes an advisory for the team that owns that service, and a guardrail doing its job needs no action.
- Simulate before shipping. Preview the fix against the real failing sessions to see whether it would have changed the outcome.
- Prove it. Verify the change against the failure's own evaluations plus the regression set built from earlier fixes. A fix without a bound proof is not marked as shipped.
- Have a person merge it. The proven change arrives as a pull request carrying its evidence. A human reviews and merges it.
- Check that it held. After the merge, compare the production volume of that failure mode before and after the deploy, and record whether the fix is verified, regressed or inconclusive.
The last piece is the eval gate: the failure mode's real examples become a regression test with a tracked baseline pass rate, run on every release. That is what turns "we fixed this" into "this cannot quietly come back."
Why the connections matter more than the layers
Each layer is useful alone, but the value is in the handoffs. Observability without evaluation gives you volume charts and no verdict. Evaluation without diagnosis gives you a long list of low scores and no priorities. Remediation without evaluation means shipping fixes you cannot prove. And a fix without a gate lasts until the next prompt edit or model change quietly undoes it.
This is the same gap described in What is an agent harness?: most of the effort goes into steering the agent before it acts, and far less into learning from what it does in production. Closing the gap does not require new instrumentation. The traces are already there. What is missing is the path from a score to a named failure, from a named failure to a proven fix, and from a proven fix to a test that keeps it fixed.
Which layers do you have?
A quick way to find the weakest connection in your own setup is to answer these in order. The first "no" is where to start.
- Can you open any production run and see each model call and tool call, with inputs, outputs and errors?
- Is a sample of live traffic scored continuously, with a trend you would notice if it dropped?
- Do you know how often your judges agree with your domain experts?
- Can you name your agent's top failure modes and point to the sessions behind each one?
- Is every fix proven against the failure before it merges, and reviewed by a person?
- After a fix ships, do you measure whether that failure actually went down?
- Does each fixed failure have a regression test that runs on every release?
Questions
Is agent observability the same as agent evaluation?
No. Observability records what the agent did: every run, step, tool call, error and cost. Evaluation reads those records and decides whether the result was good, usually with judges that score each run. You need the first to do the second.
Do I have to replace my tracing tool to add evaluation and remediation?
No. Evaluation and remediation work on the traces you already collect. Neens accepts standard OpenTelemetry and OpenInference traces, and can also pull traces from an existing tracing tool on a schedule, so you keep your current instrumentation.
How often should an agent be evaluated in production?
Continuously, on a sample of live traffic, plus on demand when you are investigating something. Scoring every trace is rarely necessary; a steady sample gives you a trend you can alert on, and deeper runs are for specific questions.
What does remediation mean for an AI agent?
A specific, tracked change that addresses a recurring failure, such as a prompt change or a tool fix, built from the traces that show the failure. Good remediation proves the change before it ships, has a person merge it, and measures afterwards whether the failure actually went down.
Where does failure clustering fit?
Between evaluation and remediation. Evaluation tells you which runs failed, one at a time. Clustering groups similar failing runs into a short list of named failure modes, which is the unit you can actually fix and gate.
Sources
- LangChain, State of Agent Engineering, n=1,340, December 2025.
- Neens documentation: Core concepts, Send traces, Judges, Failure clustering, Remediations, Eval-verified PR, Fix outcomes.