Evaluating an agent over MCP means giving a coding agent tool access to the evaluation system itself: the traces, the named failure modes, the judges, the datasets and the backlog of proposed fixes. With those tools the coding agent reads what is failing, makes the change in your repository, and asks the evaluation system to prove the change against the same evals the failure originally broke.
Why evaluation belongs in the coding agent's toolbox
The Model Context Protocol is, in the words of its own documentation, "an open-source standard for connecting AI applications to external systems." A server exposes tools; any MCP-compatible client can call them. Coding agents are MCP clients too, so a server is a natural way to hand a coding agent new abilities.
Coding agents now build and change much of an agent's harness: the prompts, the tool contracts, the routing. What they usually lack is the evidence. They can edit a system prompt in seconds, but they cannot see that the same refund failure happened hundreds of times last week, or check that their edit fixes it without breaking something that was fixed last month. That evidence lives in the evaluation system, outside the editor.
An evaluation MCP server closes that distance. The failures, the judges that recognise them and the tests that guard against them become tools the coding agent calls in the same session where it writes the code.
What an evaluation MCP server exposes
The Neens MCP server groups its tools by the job they do, in roughly the order a loop uses them. Each tool is marked read or write, and read tools carry MCP's readOnlyHint annotation so a client can approve them automatically.
| Job | What the coding agent can do | Example tools |
|---|---|---|
| Your connection | Confirm which agent this session is scoped to before reading or writing | get_current_project |
| Traces | Search runs by cluster, score, model, tool or time, and open one in detail | list_traces, get_trace |
| Failure modes | Read the named failure modes with session counts and root-cause hypotheses, and see examples of each | list_failure_modes, get_cluster_exemplars |
| Judges and scores | Find which traces a judge failed, create and deploy a judge, start an eval run | list_score_rows, create_judge, run_eval |
| Datasets | Build a dataset from selected sessions and freeze it as a golden version | create_dataset, create_golden_version |
| Labels | Record a pass or fail ground-truth label on a trace or session | add_annotation |
| Pre-prod evaluation | Run a candidate version over a golden dataset and compare runs | create_preprod_run, compare_preprod_runs |
| Fixes | Pull a proposed fix, prove it, and report where it landed | get_fix_bundle, run_verification, report_fix_status |
The last row is the one that turns investigation into repair.
The fix loop, step by step
Here is the round trip a coding agent makes to take one production failure from the backlog to a merged, verified fix. Neens does the diagnosis and the verification. Your coding agent, in your repository, with your credentials, does the edit and opens the pull request.
- Pick a fix.
list_open_remediationsreturns the open fixes ranked by priority, each with its status and any pull request already reported. - Get the brief.
get_fix_bundlereturns the fix bundle: the root cause, a typed before and after fix, anonymized failing examples, the proof-eval command, the gate policy and the acceptance criteria. - Make the change in your repo. The coding agent edits the code with its own model and pushes a branch to a preview deploy. Neens is not involved and never sees your source.
- Prove it.
run_verificationpoints Neens at the preview deploy at a version label. Neens replays the fix's golden dataset as a pre-prod evaluation, scores it with the proof judges, and gates on a minimum pass rate and a maximum number of regressions. If the verdict takes longer than the wait budget,get_verification_runpolls for it. - Report it back.
report_fix_statusrecords the pull request URL and commit SHA, so the fix is permanently linked to the code that resolved it. - After a human merges.
record_fix_mergeopens a close-out watch: Neens compares the failure mode's real production volume before and after the deploy and marks the fix verified, regressed or inconclusive.
For teams that prefer Neens to write the patch too, start_fix_run swaps out the middle steps: Neens proposes the change, verifies it with repeated pre-prod runs, and opens a pull request only if every run is green. A person still merges.
Triage is not proof
Read-only access to traces is useful. A coding agent that can list failures and open examples writes better fixes than one working from a bug report. But read access alone stops at a suggestion: the agent proposes a change and a person ships it on trust.
The difference in the loop above is step four. The fix is run against the golden dataset built from the failure's own evidence, on the preview deploy that contains the change, with a gate that fails on regressions. "Here is a suggested fix" becomes "here is a fix that passes the evals the failure originally broke," and the verdict is recorded next to the pull request where the reviewer can read it.
Step six carries the same idea past the merge. A fix that passed before release still has to show up as fewer failures in production, and the close-out watch measures exactly that.
Guardrails that keep it safe
Handing a coding agent write access to an evaluation system is only sensible if the boundaries are tight. These are the ones that matter, and how Neens draws each.
| Concern | How it is bounded |
|---|---|
| Who is acting | A per-user browser sign-in authorizes the coding agent as you, so every call is attributable to your account and limited by your role. You can view and revoke the authorization in settings. |
| What it can reach | A connection is scoped to one agent. A tool that takes an id refuses an object from a different agent. |
| What it can change | No tool deletes anything. Every call re-runs the same authorization checks as a normal API request, so a tool can only do what the credential could already do. |
| Your code | The edit happens in your environment. Neens never receives or stores repository credentials, and the fix bundle carries no keys. |
| Shipping | A human reviews and merges every pull request. Neens does not merge, approve or deploy anything. |
Getting started
- Connect the server. Add the Neens MCP endpoint to your coding agent and sign in through the browser. For headless automation with no person to sign in, use an agent API key instead.
- Check the connection. List the tools, then call
get_current_projectto confirm the session landed in the agent you expected. - Read before you write. Ask the coding agent to summarise
list_failure_modesand open a few exemplars. It is the fastest way to see what your agent gets wrong. - Close one fix end to end. Pick the top open remediation, pull its fix bundle, implement it on a branch, verify it against the preview deploy, and report the pull request back.
After one round trip the pattern is clear, and the next fix is a single prompt to the coding agent.
Questions
What is MCP?
The Model Context Protocol is an open standard for connecting AI applications to external systems. A server exposes tools; any MCP-compatible client, including a coding agent, can call them.
Does Neens see my source code?
No. Your coding agent makes the change in your repository with its own model and your credentials. Verification calls your preview deploy over HTTP, and the pull request URL and commit SHA you report are the only code references Neens keeps.
Can a coding agent delete or overwrite things in Neens?
No tool deletes anything. Write tools cover documented, non-destructive actions such as creating a dataset, recording a label or reporting a fix status, and every call runs the same authorization checks as the API, so the agent can only do what its credential could already do.
Who merges the fix?
A person. The coding agent opens the pull request in your repository and reports it back, and a human reviews and merges it. Neens does not merge, approve or deploy anything.
Do I have to share an API key with the coding agent?
Not for interactive use. A per-user browser sign-in authorizes the coding agent as you, governed by your role, and you can revoke it from settings. An agent API key remains the option for headless automation with no person to sign in.
Sources
- Model Context Protocol, What is the Model Context Protocol (MCP)?, modelcontextprotocol.io.
- Neens Docs, MCP server, Fix bundles and Eval-verified PR.