News · 2026-08-03

A New Benchmark Asks Whether a Coding Agent Can Stop Asking

A new benchmark called CAPA tests the thing users actually notice about a coding assistant with memory: after it has watched you resolve the same kind of ambiguity five times, does it finally write what you meant, or does it open the clarification loop again? Across 12 models, having the correct user's history raises first-turn success substantially, and the best performer still fails to get it right on the first response in roughly four sessions out of ten.

Key facts

• The best model tested, Claude Opus 4.8, reaches 60.3 percent first-turn success with same-user history, up from 24.3 percent without it.
• CAPA contains 600 coding sessions across 60 balanced user-and-ambiguity cells, with 300 held out for evaluation.
• Twelve models were evaluated; 11 of 12 improved eventual success with same-user history and all 12 improved first-turn success.
• Primary source: Fewer Clarifications, Better Code, arXiv:2607.26611, submitted 29 July 2026.

How it works

The construction is the clever part. Rather than collecting messy real sessions, the authors take unambiguous executable coding tasks and deliberately break them, injecting six recurring ambiguity mechanisms through a controlled three-stage pipeline and turning each into a multi-turn session. The taxonomy includes things like domain-specific words that mean two things, habitual context omission, and unstated constraints. Ten synthetic user profiles each get two of these mechanisms, consistently, so a user's ambiguity is a stable trait rather than noise.

The evaluation then runs three conditions, and the middle one is what makes the result trustworthy:

• No history. The model sees the ambiguous task cold.
• Shuffled history. The model sees the same volume of resolved coding dialogue, but from other users.
• Same-user history. The model sees this user's own prior resolved sessions.

Without the shuffled condition you cannot tell personalization from generic in-context learning, because any pile of worked examples helps. With it, you can subtract.

What the subtraction shows

The honest reading is more mixed than the headline. Shuffled history alone lifts eventual success by 10.7 points for GPT-5.5, 1.3 for DeepSeek V4 Pro, and 3.7 for GLM-5.2. So a good chunk of the benefit is just having examples.

Correct same-user history then adds its own boost, and it lands almost entirely in first-turn success: 2.0 points for GPT-5.5, 2.0 for DeepSeek, 12.0 for GLM-5.2. On eventual success, the extra gain over shuffled history is tiny or negative -- GPT-5.5 is actually 0.7 points worse with the correct user's history than with a stranger's.

That pattern is coherent, and it is the finding. Personalized history does not make models more capable. It makes them more willing to commit immediately, which is precisely the user-visible behaviour at issue. The models could already solve these problems given enough turns; the authors' control experiment confirms it, showing that with the original unambiguous specification plus debugger feedback the tested models eventually solve everything. What CAPA measures is the recovery of omitted intent and the management of the dialogue, not code generation ability.

The memory systems underperform

The result most relevant to anyone shipping an agent: purpose-built memory frameworks did worse than doing nothing clever. The authors compare mem0 and A-mem against simply passing the raw same-user history, and both underperform it for DeepSeek V4 Pro and GLM-5.2 across all three metrics, with mixed results for GPT-5.5.

Their own lightweight alternative is a gate rather than a retrieval system: check whether the history actually contains evidence of ambiguity resolution before deciding what to put in the context. That improves first-turn success for all three models while keeping eventual success roughly flat.

The lesson generalizes past this paper. Summarizing and retrieving are not the same problem as knowing when you have earned the right to stop asking. Our coverage of organising an agent's memory halving search cost without improving answers found the mirror image of this, and our agent memory lesson covers the machinery.

Why it matters

Clarification questions are the most common complaint about coding assistants that are otherwise good, and they are the reason people describe the tools as tiring. This benchmark makes that experience measurable, and it establishes a floor: even with five clean, consistent, perfectly relevant prior sessions, the strongest model available gets it right immediately only three times in five. See also coding agents that stop rediscovering the repository.

The honest caveat: this is a synthetic benchmark, and the authors are careful about it. Mechanism-level results should be read as balanced marginal associations, not causal effects of isolated mechanisms. Ten profiles with two stable ambiguities each is a much cleaner world than a real developer working across an evolving repository with changing teammates and contradictory preferences. Nothing here tests conflicting preferences, privacy, stale memories, or what it costs when an agent confidently applies a preference you have since abandoned. The paper also provides no clear public download link or repository, so the benchmark is not reproducible from the paper alone.

Primary source, verified: read the paper → (arXiv 2607.26611)

Ground Truth. (2026, August 3). A New Benchmark Asks Whether a Coding Agent Can Stop Asking. Ground Truth. https://groundtruth.day/news/a-benchmark-asks-if-a-coding-agent-can-stop-asking.html

@misc{groundtruth:a-benchmark-asks-if-a-coding-agent-can-stop-asking,
  title  = {A New Benchmark Asks Whether a Coding Agent Can Stop Asking},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/a-benchmark-asks-if-a-coding-agent-can-stop-asking.html}
}

Topics: benchmarks · coding-agents · agent-memory · personalization · evaluation · llm