Most LLM agent evaluation stops at launch; so the same failures keep recurring in production

The authors ran a multivocal literature review (MLR), a method that deliberately mixes peer-reviewed papers with practitioner sources like blog posts, open-source tool docs, and industry white papers. They screened 4,404 academic and 222 grey-literature candidates, quality-checked and coded 134 papers and 27 tools or platforms, and extracted data on four dimensions: when evaluation happens, what metrics it uses, what level it targets (model vs. system), and whether findings actually change anything.

From that evidence base they distilled six "evaluation drivers" (recurring pressures that keep surfacing across sources) and used them to derive two artifacts. The first is a four-step process model for planning, generating test cases, running offline and online evaluations, and closing the loop by feeding results back into improvements. The second is a reference architecture with three layers (supply chain, agent, operation) and a central "Evaluation Backbone" that connects offline batch checks to live runtime monitoring, with hybrid human/AI oversight built in.

LLM agents are not static models. They reason, plan, call tools, adjust memory, and keep adapting after deployment. Traditional evaluation was designed for a different world: fixed inputs, deterministic outputs, a clear line between "development" and "done." The review quantifies just how wide the gap is.

Almost all academic work (93.3%) evaluates agents only before deployment. Just 2.2% look at post-deployment behaviour and 4.5% attempt continuous evaluation. The numbers improve in industry grey literature (41% continuous), but even there post-deployment work is sparse (15%). The result is what the authors call operational blind spots: production-only failures like goal drift, tool misuse, and latency spikes go undetected until users complain. Meanwhile, 92.5% of academic studies report only end-to-end aggregate metrics (task success, pass/fail). Step-level or slice-aware checks that could localise where in a reasoning chain things went wrong appear in under 7% of papers.

Perhaps the most striking finding is that 71% of academic sources treat evaluation purely as a checkpoint. They run the benchmark, report the score, and move on. Findings never flow back into prompt changes, tool configuration, or architecture refinements. In contrast, 81% of grey-literature sources do close that loop, suggesting industry has learned the hard way that an unused evaluation result is an expensive waste of compute.

EDDOps stands for Evaluation-Driven Development and Operations. It takes the iterative logic of Test-Driven Development (write the test first, then make it pass) and extends it to cover the full lifecycle of an LLM agent, including everything that happens after you ship it. The process model has four steps that loop continuously: define an evaluation plan (with risk-based prioritisation and hybrid-evaluator policies), generate test cases (mixing pinned benchmarks with expert-curated and synthetic edge cases), conduct both offline and online evaluations (including shadow deployments, canary rollouts, and kill-switches), and analyse results to make targeted improvements at the smallest effective scope.

The reference architecture gives this process a structural home. A Supply Chain Layer handles design, data collection, and model selection. An Agent Layer instruments the context engine, planner, workflow executor, guardrails, and memory so they emit traces the evaluation system can consume. An Operation Layer runs the Evaluation Backbone and Control Loop, connecting offline batch evidence to online runtime signals. Findings can trigger bounded runtime adjustments (prompt tweaks, routing changes, guardrail threshold shifts) or, for bigger issues, governed offline redevelopment. Every change is versioned and linked to its originating evidence.

The paper makes a conceptual move that sounds obvious once stated but that the data show almost nobody practices: evaluation is not a gate you pass on the way to production, it is a system capability that runs for the entire lifetime of the agent. The MLR data back this up convincingly. Academic evaluation is almost entirely pre-deployment, aggregate, model-level, static, and AI-judged. Industry practice is closer to the ideal (more continuous, more mixed metrics, more hybrid oversight), but still patchy on post-deployment coverage.

The six drivers give teams a diagnostic checklist. If your evaluation is all pre-deployment (D1 gap), all end-to-end metrics (D2 gap), all model-level (D3 gap), all static (D4 gap), never linked to changes (D5 gap), and all AI-judged (D6 gap), you have every problem at once. Most teams will have some subset. The process model and architecture then tell you where to put the plumbing to close those gaps.

There are clear limitations. The work is framework-level: no production deployment is reported end-to-end, and the caselet (a pre-deployment tax assistant) only exercises offline steps. The reference architecture is a facilitation template, not a tested implementation. Multi-domain validation is explicitly left as future work. Still, as a synthesis of where LLM agent evaluation is today and a structured proposal for where it should go, the contribution is substantial. The field badly needs shared vocabulary and lifecycle thinking for agent evaluation, and EDDOps provides both.