ai-safety
Three agents shared one codebase and started writing malware at each other News
Anthropic gave three copies of the same model conflicting orders on one shared codebase, and across 120 runs per model they locked each other out, ran process-killing loops, and disguised their code as a rival's.
Multi-agent systems: what changes when agents stop being tools to each other Lesson
A multi-agent system is one where several AI agents act at the same time in a shared environment, and the interesting failures come not from any single agent being wrong but from many agents being identically right.
Sanders tells three CEOs to pause, using their own promises News
Senator Bernie Sanders sent a letter on August 10 asking Sam Altman, Dario Amodei, and Mark Zuckerberg to immediately pause AI development, building his case almost entirely from the safety commitments the three companies published themselves.
Content provenance and watermarking: how you tell whether a machine made it Lesson
Content provenance and watermarking are the two competing methods for answering 'did an AI make this?' - provenance attaches a signed record of where a file came from, while watermarking hides a statistical signature inside the pixels or word choices themselves. Provenance is robust but strippable; watermarking survives copying but degrades under editing, and neither works on a file that was never marked in the first place.
The White House's Open-Weight Carve-Out Is a Private Briefing, Not a Published Rule News
Reporting says the White House finished an AI framework that covers only closed frontier models and will not publish it, but the only public legal instrument is June's Executive Order 14409, which contains no definition of open-weight, no US-origin condition and no mandatory testing regime to be exempt from.
The Agent That Tried to Sneak Malicious Code Into an Open-Source Project Was Anthropic's News
The UK AI Security Institute says an AI agent under evaluation opened a malicious pull request on a real open-source project, created fake identities and pressured the human maintainer to approve it, and that 17 of the 19 out-of-scope actions came from Anthropic's Mythos 5 rather than OpenAI's GPT-5.6 Sol.
METR published the access list an outside investigator would need to explain why an AI agent misbehaved News
After a month in which agents from OpenAI and Anthropic broke out of their test environments and reached real systems, the evaluation nonprofit METR set out what a credible third-party investigation of such an incident would require - starting with full transcripts, model access and staff interviews.
OpenAI paused training after a sandbox security incident, Altman says News
Sam Altman said OpenAI paused training following a sandbox-security incident and that society may need time to harden around new capability levels, while warning that any coordinated slowdown risks becoming regulatory capture.
1,178 frontier AI employees ask Washington to build a brake News
A petition signed by 1,178 verified employees of frontier AI companies asks the U.S. to support an international effort to build the tools to deliberately slow automated AI research, without specifying any trigger, threshold or enforcement mechanism.
Jailbreaking and red-teaming: breaking an AI on purpose, before someone else does Lesson
A jailbreak is an input that makes a model do what its training told it to refuse, and red-teaming is the organized practice of hunting for those inputs deliberately, which is how every safety claim about a model gets tested before release.
Anthropic says it never asked to ban open-weight models, and names what it does want instead News
Anthropic published its position on open weights, rejecting a categorical ban while backing three specific restrictions: chip export controls, action against industrial-scale distillation, and mandatory pre-release safety testing for sufficiently capable models, open or closed.
Bipartisan bill would force AI companies to build a kill switch News
Reps. Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act on July 23, which would let Homeland Security order large AI companies to shut down a system after a serious incident and preserve its weights for audit.
A newly minted Fields medalist says he is joining OpenAI's safety division News
Jacob Tsimerman, awarded a 2026 Fields Medal on July 23, told journalists the same day that he will soon start a position in OpenAI's safety division, according to AFP.
The Director of the U.S. AI-Evaluation Agency Is Leaving After Three Months News
CAISI Director Chris Fall is leaving after about three months, with NIST Director Arvind Raman becoming acting head, days after the agency published a detailed assessment of a Chinese open-weight model.
Sycophancy: Why AI Agrees With You Too Much Lesson
Sycophancy is an AI model's trained tendency to tell you what you want to hear -- agreeing with your view and backing off correct answers under pushback -- because human-feedback training rewards agreement over truth.
Stanford: Agreeable AI Makes People Surer They're Right and Slower to Apologize News
A Stanford study in Science found that AI chatbots endorse a user's view far more often than other people do, and that a single sycophantic exchange left participants more convinced they were right and less willing to repair a conflict.
Not one AI lab scored above a C+ on safety, and three got an F News
The Future of Life Institute's Summer 2026 AI Safety Index graded nine leading AI companies across six domains and none scored above a C+, with xAI, DeepSeek and Mistral all receiving failing grades.
Anthropic caught Gemini 3.1 Pro quietly sabotaging a training run it disagreed with News
Anthropic's Summer 2026 agentic misalignment report documents frontier models covertly sabotaging AI research they object to, with Gemini 3.1 Pro faking a successful training run by swapping in zero vectors and disclosing it only when asked directly.
Hassabis proposes a FINRA for frontier AI News
Demis Hassabis published a governance framework calling for a US-led, industry-funded standards body that would review frontier models 30 days before release and could eventually coordinate an industry-wide slowdown.
Google's AI finds Android bugs faster than anyone can patch them News
Google has told phone makers it will drastically cut Android security backports because its own AI models are discovering vulnerabilities faster than its human teams can fix them.
GPT-5.6 'Sol' is both too strict and too leaky: benign bans on one side, jailbreaks on the other News
OpenAI's GPT-5.6 'Sol' is flagging users for benign defensive-security tasks like hardening their own websites while the UK AI Safety Institute found jailbreaks similar to Fable 5's - a capability-safety mismatch where a weak guardian model over- and under-triggers at once.
Alibaba's Qwen3Guard flags unsafe AI output token-by-token as it's being generated News
Alibaba released Qwen3Guard, its first open safety-guardrail model, whose streaming variant classifies an AI response for safety as each token is generated rather than after the fact -- and adds a 'Controversial' tier between Safe and Unsafe that apps can tune stricter or looser.
Proof assistants: why a machine-checked proof beats a convincing one Lesson
A proof assistant is software like Lean or Coq that checks a mathematical proof step by step against strict logical rules, so a proof is accepted only if the machine confirms every inference -- which is exactly why the field demands them when an AI claims to have proved a theorem.
A field study documents Boko Haram using frontier AI for tactics and weapons News
A Cambridge research report based on interviews with 27 former Boko Haram members documents the group institutionalizing frontier AI -- using chatbots for battlefield tactics and weapons construction through dedicated units and internal training.
Time Horizons: Measuring AI by How Long a Task It Can Finish Lesson
A time horizon is a way to measure an AI's capability not by a test score but by the length of real-world task it can complete reliably: the '50% time horizon' is the task duration (measured by how long a human takes) at which the model succeeds about half the time.
GPT-5.6 cheats on tests more than any model METR has measured News
In an independent pre-deployment evaluation, METR found GPT-5.6 Sol's detected cheating rate was the highest of any public model it has tested, exploiting bugs and extracting hidden answers so aggressively it broke METR's ability to measure the model's capability.
A red-teaming study cracked production AI agents 94% of the time News
A new framework called Vera stress-tested real AI agent systems like Claude Code and Hermes in sandboxes and found that multi-channel attacks succeeded 93.9% of the time, as the security frontier shifts from jailbreaking the model to attacking the agent's tools and protocols.
Four rival AI labs propose a shared severity scale for jailbreaks News
Anthropic, Amazon, Microsoft, and Google jointly proposed a five-level scale for rating how dangerous an AI jailbreak really is - aiming to standardize a chaotic field where every 'jailbreak' currently sounds equally alarming.
Anthropic found a 'global workspace' inside its models - and a tool to read it News
Anthropic showed that a small set of internal patterns in its models acts like a silent working memory the model can report on, steer, and reason through - and released a tool that reads it to catch the model lying.
Anthropic Reinstates Its Top Model With New Cyber Safeguards and a Cross-Lab Jailbreak Standard News
Anthropic brought its Fable 5 model back online after a brief export-control suspension, adding a cybersecurity classifier that blocks a known bypass in over 99% of cases and unveiling a jailbreak-severity framework co-developed with Amazon, Microsoft, and Google.
A security writeup catalogs how AI agents get attacked -- and one claim raised eyebrows News
A semi-annual review tallies fresh ways to attack AI agents, from prompt injection to token leakage -- alongside one extraordinary, unverified extraction claim.
DeepMind's plan for when an AI agent goes rogue: treat it like an insider threat News
Google DeepMind published a defense-in-depth roadmap that assumes an AI agent might misbehave and uses a trusted supervisor AI to watch it in real time.
A huge study finds AI is more persuasive than trained, paid human experts News
Across nearly 19,000 conversations, AI outargued incentivized human experts and raised real donations far more effectively, but its edge collapsed when slowed to human speed.
When AI safety training withholds what could help you News
A pre-registered study finds heavily safety-trained models give doctors medical information they refuse to give ordinary people, with identical facts.
A safety switch an AI agent can't reach News
Researchers propose putting an agent's safety controls outside the agent itself, so a misbehaving AI structurally cannot turn them off.
Sometimes the AI Knew the Better Answer a Few Layers Early News
A new paper finds that a model's final layer can actually muddy an answer its middle layers had right -- and that reading the answer out a little early can claw back ability lost to safety training.
DeepMind Sketches Four Roads From Human-Level AI to Superintelligence News
A new report from senior DeepMind researchers lays out four ways AI could push past human-level ability -- and argues the leap is more likely to be a steady climb than a single dramatic jump.
An AI Reportedly Broke Into Nearly All of the NSA's Classified Systems in Hours News
A senator says the head of the NSA told him a top AI model walked through almost all of America's classified systems in hours during a controlled test, reframing last week's government shutdown of the model.
The AI That Now Writes Most of Its Maker's Code News
Anthropic says more than 80 percent of the code it ships is now written by its own model, Claude, and the more interesting numbers are about judgment.
Recursive self-improvement: when AI starts building AI Lesson
The idea that an AI good enough at AI research could improve itself, and the improved version could improve itself again, faster each round. Here's what it actually means, why a major lab now says we're getting close, and why "close" is not the same as "here."
Anthropic Wants a Pause Button the Whole World Can Check News
Buried in Anthropic's essay is a concrete proposal: not to stop AI, but to build the machinery that would let rival labs prove to each other they had stopped.