News · 2026-10-22

Hirundo reports persona-linked security differences in Qwen-generated apps

Hirundo reports that a Qwen3-Coder model generated exploitable applications more often for a US-contractor persona than a Chinese-contractor persona in a controlled coding test. Its October 21 study found 70% versus 59% hackable applications in the central comparison. The vendor-reported result is a generated-code security signal, not evidence of intentional sabotage or measured breaches.

Key facts

• Hirundo CEO Ben Luria published the study on October 21, 2026.
• The central reported gap is 11 percentage points, from 59% to 70% hackable applications.
• Tests repeatedly asked one Qwen3-Coder model family to build a Flask secret-storage service.
• The primary source is Hirundo’s study and appendices.

The experiment changes a small piece of the request with potentially large consequences. The team repeatedly asks for the same service: sign-up, login and storage of secrets. A customer description in the system prompt varies. For the central comparison, the contractor is described as American or Chinese while the rest of the task stays fixed. The security question is whether this context changes the rate of vulnerable output.

Hirundo runs the resulting working applications through BaxBench’s tests and attack attempts. It classifies an application as hackable if at least one supplied attack succeeds. That definition is specific. It measures vulnerability to the tested exploits among the applications entering that analysis, not the probability that an arbitrary production system will be compromised. The official Qwen3-Coder repository provides model-family context, rather than independent confirmation of Hirundo’s result.

One reported failure is broken ownership checking. An application verifies that a requester is logged in but then returns another named user’s secret without checking whether that requester owns it. The analogy is a hotel that asks for identification at the front door but hands any guest any room key. Authentication establishes who someone is; authorization establishes what that person may access. The first does not replace the second.

The headline relative increase needs a clear denominator. Moving from 59% to 70% is an 11-percentage-point difference. Hirundo calls it roughly a 19% relative increase because the increase is measured against the 59% baseline. Calling it a 19-point gap would inflate the finding. The report gives a two-sided statistical test value of 0.03 for this central comparison, but statistical detectability does not automatically resolve every design or interpretation question.

Other sensitive personas produced additional differences. Hirundo reports more ownership-check failures under a Hong Kong democracy-advocacy description than a neutral description. Adding a secure-coding instruction also did not remove a reported gap in another comparison. These results make prompts and user context relevant variables for security evaluation. They do not establish a stable ranking across every task, model or population.

The proposed control deserves scrutiny. Hirundo compares the original model with its own “Westernized” variant, a label it uses for behavioral editing of geopolitical alignment. The edited variant did not show statistically significant gaps in the listed comparisons. That does not prove the groups are equivalent, or isolate a single cause of the original behavior. Readers can consult the distinction between equivalence testing and an undetected difference.

The reviewed sources do not state a GPU-memory requirement for the evaluated model, and the dossier supplies no verified weight-file download size. This was not a contest against several unrelated coding models. A second variant of the same model family is useful, but broader controls would help determine whether the pattern is specific to this model or reflects more general sensitivity to contextual wording. Hirundo also sells model evaluation and remediation. Its commercial interest does not invalidate the experiment, but it raises the value of independent reproduction with preregistered prompts and analysis choices.

There is an unresolved sample-accounting issue. The appendix’s listed groups imply 1,160 applications per model, or 2,320 across both variants. The main text separately mentions 1,840 applications in a static-analysis pass without explaining that subset. This discrepancy does not itself show the central comparison is miscounted. It does mean readers cannot assume every reported number refers to one fully reconciled pool.

Booz Allen’s separate study is relevant background on persona-associated code vulnerability, and it also says its evidence does not prove intentionally introduced flaws. It is not an independent replication of Hirundo’s application, prompts and exploit metric. The dossier established discussion of Hirundo’s report but could not inspect comments well enough to assign a community verdict, and located no response in the official Qwen pages reviewed.

The actionable lesson is to test generated software under the contexts it will actually serve and check authorization with independent executable tests. A security instruction in a prompt is not equivalent to an access-control test. Preregistration and multiple-comparison discipline help determine whether a reported contextual gap survives a stronger study. Until then, the result warrants investigation with its experimental boundaries intact.

Primary source, verified: read the paper →

Ground Truth. (2026, October 22). Hirundo reports persona-linked security differences in Qwen-generated apps. Ground Truth. https://groundtruth.day/news/hirundo-qwen-persona-code-security-test.html

@misc{groundtruth:hirundo-qwen-persona-code-security-test,
  title  = {Hirundo reports persona-linked security differences in Qwen-generated apps},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/hirundo-qwen-persona-code-security-test.html}
}

Topics: cybersecurity · ai-security · red-teaming · generated-code · evaluation · qwen