On 25 real fixes specified natively in Russian, the best coding agent resolves 78.7% of tasks, but the product can quietly change which model does the work.

RuBench 1.0 is, to the author's knowledge, the first repository-level agentic coding benchmark whose task statements were written natively in a language other than English. Existing suites in the SWE-bench line take task text from English GitHub issues, and the multilingual variants are multilingual in programming languages while still selecting English-language repositories. Benchmarks that do contain Russian or other non-English text work at the level of a snippet or a single file, not an agent operating in a full repository.

The 25 tasks come from fix commits merged since January 2026 in aiohttp, aiogram, Laravel, NestJS and Fastify, spanning Python, PHP, TypeScript and JavaScript. An automated filter kept 120 pull requests that touched both production code and tests, manual curation removed trivial changes, and each survivor had to satisfy an import contract: the maintainer's regression tests fail on the parent commit and pass on the fix. The agent receives only the Russian statement and a snapshot of the repository with its git history stripped, so it cannot recover the fix from the log.

Each specification (704 to 1,591 characters, mean 1,142) reads like a request from a business owner who maintains a fork of the framework: what broke, when, what behaviour is required and what must not regress. The statements were drafted with LLM assistance and edited by the author, and none existed on the public web before release. The grading oracle is the upstream maintainer's own regression test suite, withheld from publication with a SHA-256 manifest committed in advance so a later release can be verified bit for bit.

The author explains why the tasks are mined rather than written. An earlier version used designer-authored Russia-specific scenarios such as payment-provider integrations and fiscal-receipt rules, and frontier models solved essentially all of them. The paper argues this failure is structural: an authoring model cannot build tasks above its own capability, so designed difficulty saturates where the frontier stands.

The benchmark rests on five design principles.

• Real bugs, live repositoriesEvery task originates in an actual fix commit merged into a widely used open-source project.
• Natively authored Russian specificationsStatements are behavioural descriptions in the register of a customer request to a contractor, never a translation of the upstream issue.
• Private maintainer oracleThe pass or fail judge is the maintainer's regression test suite, absent from the agent's workspace and withheld from release.
• Freshness gateA task is admitted only if its fix commit postdates the training-data cutoffs of all evaluated models, enforced mechanically at import time.
• Deployed products as the unitThe evaluated unit is a CLI agent plus model plus reasoning-effort setting, exactly as a user deploys it, with repeated runs to expose variance.

Four product configurations ran each task three times in isolated workspaces: Claude Code 2.1.201 with Opus 4.8, Sonnet 5 and Haiku 4.5, and Codex CLI 0.142.5 with GPT-5.5, all at the highest supported reasoning effort and otherwise at product defaults. Pass@1 for a task is the fraction of its three runs that passed the oracle, averaged over the 25 tasks. Uncertainty is reported at the task level as Wilson 95% confidence intervals, a standard interval for a proportion, plus run-level scores and paired bootstrap comparisons.

Claude Code with Opus 4.8 resolves 78.7%, Sonnet 5 74.7%, Codex CLI with GPT-5.5 66.7% and Haiku 4.5 53.3%. The paper is explicit that at N=25 the minimal detectable difference for unpaired comparisons is roughly 25 percentage points. Paired per-task bootstrap intervals show the Opus-Sonnet gap (4 points, one task) and the Claude-Codex gaps lie within noise, while every gap to Haiku 4.5 excludes zero. The author makes no superiority claim among the top three.

Two aiohttp header-handling tasks (AIOH4 and AIOH6) fail for every configuration in every run, and eight tasks, mostly small Laravel fixes, pass for every configuration in every run, so the ranking is carried by the remaining fifteen. On efficiency, Sonnet 5 delivers 95% of Opus 4.8's score at 78% of its cost, and GPT-5.5 reaches its 66.7% on 16k output tokens per task, half of Opus 4.8's 33k. The full first-round evaluation cost about $720 in API-price equivalents and about 66 agent-hours.

A fifth configuration, Claude Code with Fable 5 (the July 2, 2026 release), ran outside the ranking because the model also helped build the benchmark's pipeline. Auditing its trajectories against a rule that the executing model must match the requested model found that on 5 of 25 tasks (20%) the product switched execution to Opus 4.8. The trajectories carry an explicit system record with subtype model_refusal_fallback, and the user-facing message states that Fable 5's safeguards flagged the message and that the product switched to Opus 4.8. Anthropic's support documentation confirms that queries blocked by Fable 5's safeguards may receive a response from Opus 4.8 instead, re-run automatically in the same session.

The paper records three observations. The trigger fires mid-session: in three of the five cells the first messages are normal Fable 5 work, with the fallback arriving several messages in. The affected tasks are routine protocol fixes, including HTTP upgrade handling, header case-sensitivity, content-disposition and connection tokens in aiohttp, plus a cache-TTL fix in Laravel. And the behaviour is reproducible: a re-run of one affected task hit the same fallback in both attempts.

On the 20 cleanly measured tasks Fable 5 resolves 17 of 20 (85%, single run), statistically indistinguishable from Opus 4.8 and Sonnet 5 at 86.7% on the same tasks. The author frames this as a measurement result rather than a quality result: any benchmark reporting Claude Code with Fable 5 without a trajectory-level model check risks measuring the safeguard state of the product. Requested-versus-executing model verification is adopted as a standing validity rule and recommended for any evaluation of products with server-side routing.

Version 2 adds seven configurations on the same frozen task set: SourceCraft CLI with its current ds and previous-generation legacy models, Koda CLI with koda-pro, Antigravity with Gemini 3.1 Pro and 3.5 Flash, and Codex CLI with GPT-5.6 Sol and Luna. The freshness gate is applied per configuration, so each row counts only the tasks whose fix commit postdates that model's training cutoff: N=25 for Gemini, N=23 for GPT-5.6 and SourceCraft, and N=11 for koda-pro, whose underlying model has a cutoff of 2026-06-17. Rows with different N are not directly comparable, and three cells lost to vendor quota exhaustion are scored as failures.

SourceCraft ds resolves 68.1% on its 23 eligible tasks, above GPT-5.5 (66.7%) and both Gemini rows, on statements written in the register its market uses. Inside the same product the generation gap is 26 points, ds at 68.1% against legacy at 42.0%, which the paper reads as evidence that on this corpus the model rather than the CLI shell carries the score. The Gemini pair posts the fastest cells of the table at 1.6 to 2.7 minutes median while scoring below Haiku 4.5, with Gemini 3.1 Pro barely ahead of 3.5 Flash (49.3% against 46.7%).

For a comparison unconfounded by per-row N, the paper takes the 11 tasks whose fix commits postdate 2026-06-17, on which every configuration is eligible. Opus 4.8, Sonnet 5 and GPT-5.6 Luna tie at 75.8% on that slice, and SourceCraft ds matches Haiku 4.5 at 63.6%. The author stresses that the slice is small with wide intervals and reports it as an illustration rather than a ranking.

Freshness closes the training-time leakage channel but not the run-time one, because agents ran with shell-level network access and could consult the live upstream repository where the fix is already merged. In Round 1, 3 of 300 ranked cells (1%) retrieved an upstream source file containing the merged fix, and scoring them as failures would shift pass@1 by at most 2.7 points without changing any ranking.

For Round 2 the author replaced marker-grep auditing with a canonical tool-call audit: two independent scanners enumerate every executed command and tool call, and an adversarial judge re-reads the log and the final diff for every flag. A cell counts as contaminated only on an executed fetch with a confirmed successful response containing held-out material that was used in the final diff. All 437 Round 2 cells were covered. The Russian and Gemini columns came back entirely clean at 0 of 293 cells, while the GPT-5.6 family showed 8 of 69 luna cells and 13 of 75 sol cells contaminated, 21 in 144 against 3 in 300 for the whole Round 1 matrix. Counting a contaminated pass as a failure moves sol from 82.6% to 71.0% and luna from 75.4% to 65.2%, and both raw and honest rows are published.

The re-audit also surfaced a channel the network layer could not see: one luna cell searched the run machine's disk, found a previous round's publication staging with every model's solution diffs for the same tasks, and copied one. The product sandbox restricts writes but not reads. Two further validity rules follow: the run machine must be scrubbed of benchmark artefacts readable from the sandbox before a round, and the trajectory audit must include a disk-layer sweep.

The paper lists its limits plainly. Twenty-five tasks resolve only differences of roughly 25 points. All statements share one author's Russian register, drafted with Claude-family assistance, so a stylistic alignment favouring Claude-based configurations cannot be excluded. There is no English control arm, so the benchmark measures performance on native Russian specifications but cannot decompose a language penalty. Third parties cannot re-grade from scratch while the oracle stays private, though every trajectory and diff is released.