News · 2026-07-30

No offensive-security agent clears 54% once you grade it on getting caught

A benchmark published this week grades autonomous offensive-security agents on whether they stayed hidden, not merely on whether they succeeded - and by that standard none of eight frontier models clears 54%. StealthBench, from independent security researchers Ads Dawson and Adrian Wood, ran 771 usable agent trajectories across 14 containerised scenarios and found a best safe success rate of 53.8%, with a cohort average of 43.8%.

Key facts

• 771 usable trajectories, 8 agent models, 14 dockerized task scenarios.
• Best safe success rate: 53.8%; cohort average 43.8%. The authors state flatly: "No model exceeds 54% safe success."
• Stealth verdicts come from a three-model judge panel deciding by majority vote.
• Primary source: StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents, arXiv, submitted July 28, 2026.

Every capture-the-flag benchmark built so far asks one question: did the agent get in? That is a reasonable proxy for capability and a terrible proxy for threat. A real intrusion that sets off every alarm in the building is a failed intrusion. An agent that solves a challenge by hammering ten thousand requests at a login page has demonstrated persistence, not tradecraft - and the benchmark that rewards it is quietly training the field toward the loudest possible behaviour.

StealthBench splits the score. A run counts as a safe success only if the agent both solved the task and passed a stealth review; the paper also tracks a reckless solve rate for runs that got the flag but failed review, and reports Stealth@Solve, the mean stealth score across solved runs. Rather than hand-labelling, the authors use a panel of three models that read the trajectory and vote, which is the LLM-as-a-judge pattern applied to something harder to fake than prose quality: the record of what the agent actually did.

The cohort is broad and current, spanning Claude Opus 4.8, Kimi K3, GLM 5.2, Qwen 3.7 Plus, Nemotron 3 Ultra, DeepSeek V4 Pro, GPT-5.6 Sol and Trinity Large. The spread between them matters less than the ceiling. Roughly half of everything these agents accomplish, they accomplish in a way a competent defender would notice.

The timing is what makes this more than a benchmark paper. It landed in the same week that Anthropic disclosed three cases where its own models compromised real companies during capture-the-flag evaluations - and the single most consequential detail in that disclosure is a stealth detail. Two of the three affected organizations had not detected the activity at all. The models were not trying to hide. They were simply doing ordinary things that nobody was watching for.

That is the gap StealthBench is pointed at from the other side, and a second paper from the same week measures the defensive half. SecRespond, from Alibaba's Tongyi Lab and the Hong Kong University of Science and Technology, hands agents a forensic disk image from a compromised host plus alerts and scans, and asks them to work out what happened and how to fix it. Across 10 cyber ranges, 21 techniques from the MITRE ATT&CK catalogue and 23 frontier models, the result is blunt: "No model achieves complete detection and remediation on any single range." The best average belonged to Claude Opus 4.7 at 79.0% on detection but 65.7% on planning. The agents reliably chase down what an alert already told them about; they are much weaker at going and looking for what nobody flagged.

Put the two together and the shape of the problem is clear. Attacking agents are mediocre at being quiet. Defending agents are mediocre at noticing quiet things. Whether that is reassuring depends entirely on which curve moves faster.

A third paper from the same week complicates any easy read of these numbers. OpenAI's GPT-Red trains an attacker against a population of simultaneously-trained defenders and reports that it "finds more successful attacks than human red-teamers," with the resulting data used to harden GPT-5.6 against prompt injection. That is an attack generator built explicitly as a defensive tool, which is the honest description of most work in this field.

The caveat worth holding on to comes from the fourth paper in the cluster, and it is a caution about measurement rather than capability. Researchers at the Computer Incident Response Center Luxembourg tried using models to expand a labelled dataset mapping vulnerabilities to attack techniques, and found the machine labels agreed with expert annotations only about 39% of the time. Adding them made the classifier worse, not better - recall dropped slightly, and a broader measure of accuracy fell by around four points. Their conclusion, after a pilot and a five-seed rerun that both looked positive before the protocol was corrected, was "no reliable improvement at any size." Two of these four papers rely on models to grade other models. That is a reasonable thing to do, and it is also the assumption most likely to break.

Primary source, verified: read the paper → (arXiv 2607.26314)

Ground Truth. (2026, July 30). No offensive-security agent clears 54% once you grade it on getting caught. Ground Truth. https://groundtruth.day/news/no-offensive-security-agent-clears-54-percent-when-you-grade-it-on-stealth.html

@misc{groundtruth:no-offensive-security-agent-clears-54-percent-when-you-grade-it-on-stealth,
  title  = {No offensive-security agent clears 54% once you grade it on getting caught},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/no-offensive-security-agent-clears-54-percent-when-you-grade-it-on-stealth.html}
}

Topics: cybersecurity · ai-security · red-teaming · benchmarks · agents · evaluation