News · 2026-11-01

Memdoor reports 49 successful coding runs from a role-based Qwen workflow on ten problems

Memdoor’s developer reports that a role-based Qwen3.8-27B workflow passed 49 of 50 attempts across ten hard programming problems. The public artifacts make the result unusually inspectable, but the run used hosted inference, repeated a small problem set and spent substantial extra computation; it does not establish frontier-level local coding at a lower cost.

Key facts

• Ten distinct programming problems were attempted five times, yielding 49 reported successes in 50 attempts.
• The developer reports $35.79 for the counted runs and a median 40 minutes of model time per problem-run.
• The November 1 discussion concerns an author-run reproduction, not an independent evaluation.
• The primary record is Memdoor’s workflow README, supported by its run notes.

The compelling idea is organizational: a model that struggles alone may benefit from a process that writes down a plan, divides the problem and checks executable work. That is different from asking five models for answers and picking the most popular response.

The current example in the Memdoor repository uses a planner, two parallel ideators, a solver and a manager or reviewer, plus a conditional repair worker. Calls exchange files containing plans, proposals, code and test results. Fresh assignments receive selected state rather than an ever-growing shared conversation. Ordinary commands run samples and decide whether the final output passes the workflow’s checks.

Think of a small engineering team with a project notebook. One person defines the approach, two suggest alternatives, another implements, and a reviewer inspects the result. The notebook preserves decisions when people change shifts. Here the roles are model calls, and the files preserve progress when a reasoning session ends or drifts. Our agent-harness lesson explains why that surrounding software can change apparent model performance.

The evaluated reproduction must also be separated from today’s copyable example. Memdoor matched the prompts and loop of the GVS5H study, including up to ten manager-worker rounds. It checked request parity against the authors’ harness in four cases. That establishes an implementation check, not a replication of every result in the original paper.

Memdoor’s selected ten-problem slice produced a mean of 9.8 successes per ten, compared with 9.6 for the source harness and 8.0 for Fable 5’s single-call results imported from the study. Fable was not rerun. The README calls 9.8 and 9.6 “the same number” given the small sample. Those five repeats are separate single-attempt evaluations, not a best-of-five selection that keeps only the strongest answer.

The statistical boundary is essential. Fifty attempts on ten fixed questions are not fifty independent questions. Repeating a familiar puzzle measures whether a method can solve that puzzle reliably; it does not show how often it solves a new puzzle. The contests date to spring 2025, and the developer warns that training exposure cannot be ruled out. Claims that the entire set postdated model training are unsupported; see benchmark contamination.

The run notes report 15.3 million generated tokens. Including interrupted or stopped attempts takes the logged bill to $38.77. Some problem-runs were much longer than the median. Model time is not a measured household wall-clock time or an electricity bill, particularly when role branches overlap and hosted providers vary.

Qwen is an open-weight model, but this experiment does not supply a verified weight-download size or a minimum graphics-card memory requirement for reproducing it locally. Those values remain unstated in the reviewed run evidence. The hosted result cannot establish that a particular consumer graphics card is sufficient, and parameter count alone would not answer that question.

The records also reveal a concrete failure. A manager invented an incorrect counterexample, persuaded the team to replace a correct program, and accepted a wrong replacement that passed public samples. The original study separately disclosed evaluator bugs and corrected stored grades. Memdoor says it used that corrected grader, so those bugs do not automatically invalidate its score.

The Hacker News discussion mixes enthusiasm with objections about token budgets, contamination, the tiny slice and comparisons with single-call baselines. No independent replication appears in the reviewed record. A useful next comparison would hold the task set, serving configuration and spending budget fixed, then compare the workflows on previously unseen problems. That would test a different claim from repeating this selected slice.

The practical takeaway is promising but bounded: organized, asynchronous model work can rescue difficult tasks, while more roles and more reasoning still require a trustworthy way to tell whether the answer is right.

Primary source, verified: read the paper →

Ground Truth. (2026, November 1). Memdoor reports 49 successful coding runs from a role-based Qwen workflow on ten problems. Ground Truth. https://groundtruth.day/news/memdoor-qwen-team-ten-problem-coding-result.html

@misc{groundtruth:memdoor-qwen-team-ten-problem-coding-result,
  title  = {Memdoor reports 49 successful coding runs from a role-based Qwen workflow on ten problems},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/news/memdoor-qwen-team-ten-problem-coding-result.html}
}

Topics: coding-agents · orchestration · evaluation · open-models