A coding agent can sound convincing after reading the wrong files. That makes demos a weak way to evaluate repository research.
We built RepoContextBench to test a narrower question: can an agent find enough of the right repository context to answer a practical engineering question accurately?
Version 1 is now public. It contains the dataset, gold answers, runner, judge contract, task-level results, tool trajectories, and an interactive dashboard. This report covers the first 41 complete runs published with it.
What the benchmark measures
RepoContextBench v1 uses one snapshot of microsoft/agent-framework pinned to commit 47fa59f8e9d7b91e382834b42ecff45e22e2d890.
The dataset contains:
- 20 practical questions;
- 82 atomic gold claims;
- 99 source-evidence spans;
- 9 regression cases for the judge.
One question is:
How would you run several specialist agents in parallel on the same request and then combine their answers into one response?
Its gold answer is not one string to match. It is a checklist of four weighted claims: broadcasting work to participants, wiring fan-out and fan-in edges, exposing the workflow as an agent, and the alternative asyncio.gather pattern shown in the repository. Each claim points to exact source spans.
The answerer receives the question as written and researches the repository. A fixed Codex CLI gpt-5.5 judge at high reasoning effort then checks claim coverage, faithfulness, evidence use, contradictions, and unsupported concrete claims.
Quality gives partial credit. Certification is stricter: an answer passes only if answerability is correct, every critical and required claim is covered, evidence use is at least 75%, and the judge finds no fabricated concrete statements or contradictions. Provider, network, harness, and judge failures are run-health failures, not zero-quality answers. Official results require all 20 tasks and zero such failures.
Judge time, tokens, and cost are recorded separately and excluded from all answerer efficiency numbers.
Quality and certification are far apart
As of July 17, the best published quality score is 80.7% from Claude Fable 5 at max effort through Claude Code. Its certification rate is 25%.
That gap is the useful result. The best system often covered most of the answer while still missing a required claim, weakening its evidence, or adding a concrete statement the judge could not support. A single pass rate would hide that distinction; a single partial-credit score would hide it in the other direction.
These rows compare complete systems, not bare models. Harness, tools, reasoning effort, provider, and research mode all affect the result.
The chart uses a log cost axis. Each point is a model, harness, and configuration. The cost axis is run cost divided by 20 evaluated tasks. The dashed Pareto line marks the best observed quality at each cost level; it does not explain why systems land where they do. The 16 runs without cost data are omitted, so the chart shows 25 cost-bearing runs. Open the full-size chart. Source data.
Controlled comparisons are more useful than the ranking
Matched pairs answer engineering questions better than the leaderboard.
What changed when semantic search was enabled
We have one matched pair where the model, harness, research mode, and every other setting are held constant. The only treatment is semantic search.
With Gemini 3.5 Flash at medium effort, enabling semantic search changed quality from 63.05% to 66.10% (+3.05 pp). Total tokens fell from 2.838M to 2.389M (-15.8%), and estimated run cost fell from $3.176 to $2.931 (-7.7%). Average task time rose from 36.14 to 38.66 seconds (+7.0%).
Open the full-size comparison.
Each side of this pair is one run, so the 3.05-point difference may sit within run-to-run and judge variance. This is directional evidence for that configuration, not a universal semantic-search multiplier or a stable effect-size estimate. It shows the trade-off we wanted the benchmark to expose: retrieval can reduce context volume and improve the answer at the same time, while still adding latency.
What changed in deep mode
For Qwen 3.6 35B A3B on DeepInfra, moving the same CodeAlive agent from standard to deep mode changed quality from 72.75% to 77.60% (+4.85 pp) and certification from 5% to 15%.
Estimated cost rose by 7.5%, total tokens by 4.1%, and average task time by 17.6%. This pair favors deep mode, but one observation per side does not establish a stable gain.
What we still cannot claim
RepoContextBench v1 is intentionally auditable, but narrow.
- It covers one repository snapshot and 20 tasks.
- All gold questions, answers, claims, and evidence are public. That makes errors easy to challenge, but allows contamination and benchmark-specific tuning.
- The benchmark measures static repository research. Agents cannot use the internet, execute code or tests, modify the repository, or verify runtime behavior.
- An LLM judge introduces residual variance even with a frozen prompt, schema, model, and regression suite.
- One run is one observation. Provider aliases, serving stacks, and prices change.
- We built both the benchmark and the CodeAlive agent that appears on its leaderboard. Public trajectories, the judge contract, and the regression suite mitigate that conflict; inspect our rows rather than taking them on trust.
For those reasons, the current results are evidence about these systems under one published contract. They are not a universal ranking of models or coding agents.
The dataset and runner, methodology, and full run dashboard are public so the tasks, trajectories, and scores can be inspected before the ranking is trusted.