News · 2026-10-04

Benzi reports 391 fixes in 500 coding tasks with a compiler-backed agent

Benzi’s developers report resolving 391 of 500 tasks in a verified coding benchmark using an agent backed by a compiler-generated code map, with $37.33 in model-generation charges. The result is an author-run evaluation of a complete agent setup, including localization and a capped review pass; it does not establish that compiler indexing alone caused the score or halves real-world development costs.

Key facts

• The published result is 391 of 500 resolved issues, or 78.2%, with one benchmark attempt per issue.
• Variant Technologies’ Benzi report lists DeepSeek v4-flash as the configured model and $37.33 in total generation cost.
• The project surfaced in the October 4 research intake; its live documentation does not establish a new launch date.
• Primary source: Benzi’s published benchmark report.

A coding agent often spends much of its time discovering where things are. It reads a file, searches for a name, follows an import, and repeats. That exploration can be valuable, but repeated rediscovery uses context and paid model calls. Benzi’s proposition is to give the agent a structured map before asking it to reason through a bug.

The Benzi repository describes a parser-based compiler that turns source code into relationships: symbols and references, inheritance, call edges, and data or control flow. Rather than asking the model to infer every relationship from a long text dump, the system exposes targeted queries against the map. One named tool is “get_callers,” which asks where a function is invoked.

Think of the difference between exploring a city from disconnected photographs and navigating with a street map. Photographs remain essential when you need to inspect a building. The map tells you which streets connect and where to look first. A compiler index can play that organizing role for source code while the model still reads the relevant implementation before proposing a change.

The distinction also defines the limitation. Static analysis reads program structure without executing every possible behavior. Dynamic dispatch, runtime values, generated code, and ambiguous relationships can leave edges unresolved. Benzi says it retains candidate or unknown edges rather than silently treating guesses as facts, and uses a runtime tracer when static information is insufficient. A map with marked uncertainty can be more useful than a complete-looking map built from invented connections.

The project describes parser-gated edits as another part of the workflow. Checking whether an edit remains structurally valid can catch a class of mistakes before a full test run. It cannot establish that the new program does the right thing. Syntax and structure provide constraints; behavioral tests and the task’s actual requirements still determine whether a fix is correct.

The benchmark report evaluates the complete bundle. Alongside the code index, the configuration includes deterministic symptom mapping, scope information at edit time, and a second model pass that critiques the result and may request one capped correction. The authors say that review requests a revision on nearly every item. One attempt per benchmark issue therefore does not mean one model call, one draft, or no feedback.

This is the same measurement problem covered in agent harnesses and scaffolding. A score is produced by a model plus an interface, search process, memory policy, editing tools, and review procedure. Benzi’s public description is useful because it names several of those components. A causal claim about the compiler’s contribution would require a comparison that changes the compiler while holding the rest of the setup suitably fixed.

The report describes official per-instance container grading and one issue without a patch. Its generation-cost figure works out to less than ten cents per resolved issue, but that is a narrow accounting measure. It is not a published price for buying a complete fix, nor a claim about all computing, setup, maintenance, and review costs. The configured model’s prices and the benchmark workflow also need to remain part of any fair comparison.

A separate lead claimed the system was twice as cheap as alternatives. The dossier could not retrieve the primary comparison page, so that numeric comparison is omitted. The verified result supports an economical author-run benchmark configuration. It does not support an independently reproduced superiority claim over every competing agent or workflow.

Privacy has a similarly precise boundary. The documented editor, command-line, and tool-server integrations keep the compiler index on the developer’s machine. Source snippets actually read by the model still go to the configured provider. A separate browser demonstration runs on the server. Local indexing changes where structural analysis happens; it does not make a remotely served language model local.

The strongest counterargument is generalization. Fixing a curated set of repository issues under a known grading procedure is different from handling an unfamiliar company codebase, ambiguous requirements, unsupported language features, and human review. The dossier found no independent replication or broad adoption evidence for this result. Its source pages are mutable, so their current contents should not be treated as a dated launch announcement.

The immediate value is a usable implementation of a clear engineering idea: let conventional program analysis answer structural questions, then spend model reasoning on interpretation and changes. Readers can examine the repository and the full protocol rather than accept the score alone. The meaningful next evidence is reproducible comparison on additional projects, including failures and total operating cost.

Primary source, verified: read the paper →

Ground Truth. (2026, October 4). Benzi reports 391 fixes in 500 coding tasks with a compiler-backed agent. Ground Truth. https://groundtruth.day/news/benzi-compiler-backed-agent-reports-391-benchmark-fixes.html

@misc{groundtruth:benzi-compiler-backed-agent-reports-391-benchmark-fixes,
  title  = {Benzi reports 391 fixes in 500 coding tasks with a compiler-backed agent},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/benzi-compiler-backed-agent-reports-391-benchmark-fixes.html}
}

Topics: coding-agents · developer-tools · benchmarks · static-analysis · open-source