Running agents in production.

Agentic Harness

What a harness is, and the layer most harnesses are missing.

What is an agent harness? The half most harnesses are missingAn agent harness is everything around the model that turns it into an agent: prompts, tools, memory, orchestration and checks. Here is what a harness contains, how guides and sensors work, and the production feedback loop most harnesses are missing. Agent observability vs evaluation vs remediation: what each layer doesAgent observability records what an agent did, evaluation judges whether it was good, and remediation fixes the failure and proves the fix held. What each layer answers, what it needs, and where most tooling stops.

Close the loop

Turning production failures into tests, and closing the loop from a coding agent.

AI agent regression testing: turning production failures into testsAI agent regression testing checks every new agent version against the failures it has already had. How to turn real production failures into regression tests: named failure modes, evidence datasets, judges checked against experts, and a baseline pass rate on every release. Evaluating and fixing agents over MCP: closing the loop from your coding agentAn evaluation MCP server gives your coding agent the traces, failure modes, judges and fix backlog as tools, so it can read what is failing, change the code in your repo, and prove the change against the evals the failure broke. How the loop works and the guardrails that keep it safe.

Measure

Choosing a model on evidence instead of habit.

Switching to a cheaper model without regressionsHow to prove a cheaper LLM is good enough for an agent: replay one frozen suite against every candidate, score with the same judges against the same bar, require every repeat to pass, and read the verdict per agent with the sample size shown.

Govern

Judges your experts can trust, and gates that hold a release to them.

LLM-as-a-judge: how to know your judge agrees with your expertsLLM-as-a-judge uses a language model to grade an agent's output against a rubric. Here is what a judge is made of, how to measure its agreement with human experts using precision, recall and Cohen's kappa, and how to calibrate it until you can trust it. Eval gates: release gates for AI agentsAn eval gate is a regression test built from one of your agent's own confirmed failures and used as a release guard. What a gate is made of, how it is built from a failure, how it differs from a pre-production evaluation, and where gates sit in the release lifecycle.

Reference

The vocabulary, defined in a sentence or two.

Agent reliability glossaryPlain definitions of the terms used in AI agent reliability: agent harness, trace, session, LLM-as-a-judge, judge alignment, failure mode, regression set, eval gate, pass^k and more, with how each works in Neens.