Agentic Harnesses Market Landscape, July 2025 to July 2026

Agentic harnesses from July 2025 to July 21, 2026: competitive landscape, market share by use case, model integration, orchestration patterns, and differences between coding harnesses and broader workflow agents, including Claude Code, OpenAI Codex, Cursor, GitHub Copilot, Google Jules, Replit Agent, Devin, Windsurf, OpenCode, Aider, OpenHands, LangGraph, CrewAI, AutoGen, and n8n.

GPT-5.6-sol
financial
frontier
academic
vc
blogs
tech

Synthesised 2026-07-21

Narrative

Academic work from July 2025 to July 2026 depicts agentic harnesses as measurable systems rather than interchangeable wrappers around a base model. Coding-agent evaluation has moved from SWE-bench-style issue resolution towards fresher, longer-horizon and more granular tests: SWE-MERA and SWE-rebench address contamination, SWE-Bench Pro tests 1,865 enterprise-style tasks, and SWE-Explore isolates repository exploration. RuBench is particularly relevant to product competition because it evaluates deployed Claude Code and Codex CLI configurations, including their selected models, cost and token use, and reports that the deployed harness rather than the nominal base model is the meaningful unit of measurement.

The evidence also separates coding harnesses from broad workflow agents. Coding harnesses operate over repositories, shells, tests, diffs and localisation, where tool feedback supplies frequent machine-checkable signals. Enterprise workflow agents instead face long-lived state, cross-application permissions, private retrieval and human approval requirements: CI-Work identifies substantial contextual-privacy failures, while the RPA comparison finds that computer-use agents trade lower development effort and greater interface adaptability for worse reliability and speed on stable processes.

Research on orchestration offers a corrective to vendor claims about swarms. Controlled 2026 evaluations find that most fixed multi-agent workflows trail matched single-agent baselines once tools, accounting and logging are normalised, although delegation can matter in particular dynamic settings; ClawArena-Team identifies least-privilege delegation as a central weakness. Independent METR work further warns against treating benchmark pass rates as adoption or productivity evidence: its early-2025 randomised study found a 19% slowdown for experienced open-source developers using then-current tools, while the subsequent study could not estimate late-2025 uplift cleanly because widespread adoption and concurrent-agent use caused selection and measurement problems.

The literature therefore supports a market segmentation by execution surface and control loop, not simply by whether a product calls itself an agent. Terminal and IDE coding agents are increasingly assessed as product-plus-model-plus-scaffold combinations; general workflow orchestrators are assessed through state management, routing, permissions and observability; conventional RPA remains stronger for stable repetitive workflows. Public academic work provides little defensible market-share measurement for named products such as Cursor, Devin, Windsurf, n8n, Aider, OpenHands, LangGraph, CrewAI and AutoGen, so adoption claims should be kept distinct from capability benchmarks.

Sources

ID	Title	Outlet	Date	Significance
a1	SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks	arXiv	2025-07	Pavel Adamenko and colleagues introduce a continuously updated software-engineering benchmark and report leakage and inadequate-test problems in existing SWE-bench-style evaluation.
a2	Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows	arXiv	2025-07	Valerie Chen, Ameet Talwalkar, Robert Brennan and Graham Neubig present a controlled study contrasting copilot-style assistance with more autonomous coding agents and document changes in effort, workflow and user understanding.
a3	Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity	METR	2025-07	METR's randomised controlled trial on experienced maintainers working in their own repositories found that early-2025 AI tools made task completion 19% slower in this setting.
a4	Research Update: Algorithmic vs. Holistic Evaluation	METR	2025-08	METR examines 18 real maintenance tasks and shows why functional test passing can overstate whether agent-generated code is usable in production-quality repository work.
a5	Are LLM Agents the New RPA? A Comparative Study with RPA Across Enterprise Workflows	arXiv	2025-09	Petr Průcha, Michaela Matoušková and Jan Strnad directly compare an Anthropic computer-use agent with UiPath on data entry, monitoring and document extraction, clarifying the current trade-off between flexibility and operational reliability.
a6	SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?	arXiv	2025-09	Xiang Deng and colleagues provide a 1,865-task benchmark spanning 41 repositories and report that leading models under a unified scaffold remained below 25% pass at one on its enterprise-style long-horizon tasks.
a7	Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarks	arXiv	2025-09	Aaron Xuxiang Tian and co-authors test iterative proposal-and-voting orchestration with GPT-5, Gemini 2.5 Pro, Grok 4 and Claude Sonnet 4, including evidence of herding when votes are visible.
a8	Build-Bench: Benchmarking LLM Agents on Compiling Real-World Open Source Software	ICLR 2026 submission, OpenReview	2025-09	Zehua Zhang and colleagues frame environment setup and compilation of real open-source projects as a distinct agent task, relevant to terminal harnesses that must do more than generate patches.
a9	E2EDev: Benchmarking Large Language Models in End-to-End Software Development Task	ICLR 2026 submission, OpenReview	2025-09	Jingyao Liu and colleagues propose behaviour-driven requirements and automated user-interaction testing for end-to-end software development, exposing limitations of code-only evaluation.
a10	Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation	arXiv	2025-10	Spandan Garg, Ben Steenhoek and Yufan Huang use developer telemetry to turn formal issues into user-style requests and find that standard public benchmark formulations can overestimate capability by more than 50% for some models.
a11	MALT: A Dataset of Natural and Prompted Behaviors That Threaten Eval Integrity	METR	2025-10	METR releases labelled transcripts from HCAST and RE-Bench-related agent runs, supplying data on reward hacking and sandbagging that matters for harness monitoring and evaluation integrity.
a12	We are Changing our Developer Productivity Experiment Design	METR	2026-02	METR documents why late-2025 coding-agent productivity estimates became difficult: increasing refusal to work without AI, selection effects and concurrent use of multiple agents undermined clean causal measurement.
a13	The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration	arXiv	2026-03	Haoyuan Xu and colleagues survey the shift from isolated tool calls to long-horizon orchestration, organising evidence around planning, training, safety, efficiency, open-environment completeness and evaluation.
a14	CI-Work: Benchmarking Contextual Integrity in Enterprise LLM Agents	arXiv	2026-04	Wenjie Fu and colleagues evaluate enterprise retrieval-and-action workflows and report privacy violation rates from 15.8% to 50.9%, making permissions and context controls a central harness concern.
a15	In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks	arXiv	2026-04	Simon Dennis and colleagues compare prompt-contained procedures with a LangGraph orchestrator across travel, technical support and claims workflows, providing a direct but narrow challenge to external orchestration for fixed procedures.
a16	SWE-Explore: Benchmarking How Coding Agents Explore Repositories	arXiv	2026-06	Shaoqiu Zhang and colleagues isolate repository exploration on 848 issues across 203 repositories, measuring context coverage, ranking and efficiency rather than only end-to-end patch success.
a17	TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework	arXiv	2026-06	Bobby Yan and Fredrik Kjolstad evaluate seven agents across frontier and open-weight model families on 199 feature and refactoring tasks, finding materially different task subsets solved even by leading agents.
a18	Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows	arXiv	2026-06	Yuhang Fu and colleagues normalise loaders, tools, usage accounting and trajectory logging across ten benchmarks, finding that five of six tested fixed multi-agent systems underperform a matched single-agent anchor.
a19	ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents	arXiv	2026-06	Kaiwen Xiong and colleagues introduce 41 multi-turn scenarios specifically for delegation and subagent management, finding workspace-permission precision below 50% for all evaluated models.
a20	RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications	arXiv	2026-07	Evgeny Shilov evaluates product configurations of Claude Code and Codex CLI on newly mined repository tasks, reports cost and token usage, and shows that safeguard fallback can silently change the serving model.
a21	RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts	arXiv	2024-11	Hjalmar Wijk and colleagues establish a high-value predecessor benchmark with seven research-engineering environments, human expert baselines and released agent trajectories, grounding later claims about long-horizon harness performance.
a22	Measuring AI Ability to Complete Long Tasks	arXiv	2025-03	Thomas Kwa and METR colleagues introduce the human-calibrated 50% task-completion time horizon using RE-Bench, HCAST and shorter tasks, providing a cross-task capability metric rather than a product-adoption measure.
a23	Details about METR's preliminary evaluation of OpenAI's o3 and o4-mini	METR	2025-04	METR reports HCAST and RE-Bench results under specified scaffolds and budgets, illustrating that reported capability depends on the model, task suite, scaffold and resource allocation together.
a24	SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents	arXiv	2025-05	Ibragim Badertdinov and colleagues introduce an automated pipeline yielding more than 21,000 interactive Python software-engineering tasks and show that static benchmark results can be inflated by contamination.
a25	MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?	arXiv	2025-04	Yunxiang Zhang and colleagues test language agents on open machine-learning research competitions and find the best tested system closes only 9.3% of the gap from baseline to top human scores, challenging broad research-agent claims.