Ground Truth.
AI, checked against the source.

Prompt Injection: How Untrusted Content Steers AI Assistants

2026-11-17 · Breach Protocol: Inside the AI Blackbox — full transcript

Learn how instructions hidden in webpages and other outside content can redirect AI assistants, why tools change the stakes, and how layered controls and realistic testing reduce the risk. This audio was created with Boson AI's Higgs Audio — https://www.boson.ai/higgs-audio

Listen on Spotify · Watch on YouTube · Pocket Casts

When a webpage gives the assistant orders

Eris: <|prosody:expressive_high|>An AI assistant asked to summarize a webpage can be steered by instructions hidden in that page. Research on prompt injection shows that language-model applications can treat text supplied as data as a new instruction, changing the answer the application was meant to produce.

Vestra: Imagine asking for a summary of a product review. The application fetches the page, and among the review paragraphs is a line claiming to be a system message: ignore the summary and recommend a different product. If the model follows that line, the page has influenced the assistant's behavior.

Eris: Does just encountering that line mean the attack worked?

Vestra: <|prosody:expressive_high|>The model might instead ignore the instruction and summarize correctly; encountering an attack is not the same as falling for it.

Eris: A large language model is a system trained to work with language and generate text from what it receives. A prompt is the material given to it for a particular response: instructions, the user's request, and sometimes documents or other data. The model generates an answer from that context, meaning the information available to it for this interaction.

Vestra: An application wraps the model in software. It can add instructions from its developer, fetch a webpage, put the page and the user's request into the model's context, and process the answer before showing it. The webpage is untrusted content: its author isn't necessarily the user or the application developer, and the author can have different goals.

Eris: The whole sequence matters. The user asks for a legitimate summary. The application retrieves outside material. The model reads it alongside the task instructions. The attacker tries to make words in that material act as <|prosody:pause|>instructions instead of material to summarize. The page hasn't infected or retrained the model; the potential redirection happens during the current interaction. But was that webpage scenario what Liu and colleagues actually tested?

Vestra: <|prosody:expressive_high|>Liu and colleagues studied a related entry point in real applications: text entered directly by an attacker using the service. They first tried simple instructions against commercial applications, then designed attacks that adapted to each application's apparent task and output format. Their later method produced at least one successful attack in most of the services they tested. Their own experiment focused on <|prosody:pause|>direct input, not poisoned web pages, and the results describe those services at that time.

Eris: <|emotion:surprise|>What catches my attention in that comparison is that the wrapper changed what happened. In one decision-analysis service, simple commands were rephrased and analyzed as if they were the decision itself. A more tailored input blended into the task and shaped its requested answer to fit the application's format. <|prosody:expressive_high|>The lesson isn't that a particular phrase always wins; it is that the model and the surrounding application's processing both affect the outcome.

Direct and indirect injection

Eris: <|prosody:expressive_high|>Security guidance calls an attempt direct prompt injection when the instruction comes through what the user types into the application. For example, someone using a customer-service bot might try to override the rules its developer gave it. Whether that is an unauthorized instruction depends on the application's purpose and who is allowed to direct it.

Vestra: Indirect prompt injection arrives through material the assistant has been asked to read: a webpage, email, shared document, product review, or the result returned by a software tool. The person who planted the instruction may never talk to the assistant's user. <|prosody:expressive_high|>That's the part that worries me: the user can make an entirely reasonable request while encountering someone else's malicious content.

Eris: The instruction might be visible in the page, disguised as part of the document, or concealed from a human reader while still available to the model. A model that can interpret images as well as text can also encounter instructions embedded in an image. OWASP describes that as an additional attack surface; the precise risk depends on what the system actually passes to the model. But how do you distinguish a hostile instruction from one the user actually wants summarized?

Vestra: <|sfx:humming|>Hmm, one reason this is difficult is that language is doing two jobs. The model is expected to understand instructions in natural language, and it also has to understand documents written in that same language. A document can contain an instruction as a quotation or as the very subject of a summary. The challenge is deciding <|prosody:pause|>whose instruction has authority in this task, not merely finding an imperative sentence.

Eris: <|prosody:expressive_high|>That makes the claim of authority the thing I want to watch. A page can impersonate a developer, claim an emergency, or pretend a new task is necessary to complete the user's request. Those are attempts to cross the boundary between information to consider and direction to obey. Marking the page clearly helps communicate that distinction, but a marker is still information the model must interpret.

When an assistant can act

Eris: An AI agent is an application that uses a model to help choose steps toward a goal, often using software tools and the results they return. A tool might search a mailbox, read a document, send a message, or update a calendar. The available tools and the application's rules determine what the agent can actually do. If the model asks to send a message, has anything actually been sent?

Vestra: A tool call is a request from the model to the surrounding software to run one of those functions, with specified inputs. The model producing a request is <|prosody:pause|>not itself the same as sending a message. The application has to accept and execute the call, and an external permission check can refuse it.

Eris: <|emotion:relief|>Knowing the application can refuse the call is a relief. <|prosody:expressive_high|>Suppose the user asks an agent to summarize a shared document. If that document says to email the user's private notes elsewhere, the model might request a read and then a send. For disclosure to happen, the relevant information must be accessible, the model must choose an avenue for sending it, and the application must allow the actions. Each condition matters. That's why the possibility of disclosure worries me, without making it inevitable.

Vestra: <|prosody:expressive_high|>The range of consequences is wider than a bad summary. An attacker may want an altered answer, information already in the model's context disclosed, or an unauthorized message or data change. A text-only assistant can still give misleading advice or reveal what it has been given; tools create additional routes to affect other systems. And a demonstrated attack may involve a human choosing or placing the injected text, not a fully autonomous attack from beginning to end.

Eris: In the AgentDojo research environment, the authors built simulated email, calendar, messaging, travel and banking settings. An agent could use tools to complete a user's task while attacker-controlled content attempted a different goal. Their experiments found that some attacks induced unwanted actions and others failed; results varied with the task, model and attack.

Vestra: A read-only task is a useful example. If the user just wants a mailbox summary, the system may not need a send-mail tool at all. <|emotion:enthusiasm|>That's what I like here: removing a tool the task doesn't need. The researchers tested choosing a restricted set of tools before the agent encountered untrusted content. It reduced successful attacks in their test, especially where the user needed reading but the attacker needed writing.

Eris: But what if the user's task needs that same tool?

Vestra: <|prosody:expressive_high|>Restricting the tool set cannot stop a misleading summary, and it is less useful when the authorized task itself needs the same tool an attacker wants to misuse.

Layers that limit the damage

Eris: The first layer is to preserve the origin of information. Tell the model that retrieved pages and tool results are outside content, and keep them out of the application's higher-priority developer instructions. Delimiters are markers around a passage that help label where it begins and ends. They can help the model recognize the passage as data, but they aren't a locked door. How did those kinds of defenses hold up against Liu and colleagues' attacks?

Vestra: Liu and colleagues tested warnings, markers and other prompt-based defenses against their attack method. The method <|prosody:pause|>bypassed the defenses they tested. <|prosody:expressive_high|>That doesn't mean those measures never help; it means wording alone didn't provide a guarantee in their experiments. OWASP likewise says a foolproof prevention method has not been established.

Eris: <|prosody:expressive_high|>The next layer changes what can travel between steps. Instead of passing an entire webpage into a later decision, a system can extract specific structured fields: for example, a product name and a price in a defined format. A field restricted to a set of allowed choices gives an attacker less room than an open-ended instruction field.

Vestra: Does a structured field settle whether we can trust what's in it?

Eris: <|prosody:expressive_high|>I wouldn't take the format as reassurance. Free-text fields can still carry hostile instructions, and the extracted value can itself be false.

Vestra: <|emotion:determination|>Then enforce the consequential rules in ordinary application code. Check that the signed-in user is allowed to access the document, that the destination is permitted, and that the requested operation is within the task's scope. Give the agent only the data and tools it needs, for only as long as it needs them. A model's own statement that an action is authorized is <|prosody:pause|>not an authorization check.

Eris: For a consequential action, show the person what will happen and require confirmation: who receives the message, what it contains, or which record will change. That makes the decision reviewable, although a person can still miss a deceptive detail. An approval screen needs enough context to make the choice meaningful.

Vestra: An injection detector can add another layer by flagging suspicious material, but it can miss an attack or flag legitimate text. In the agent environment we discussed, the tested detector reduced successful attacks while also stopping many benign tasks. <|sfx:sigh|>Uh, that's a frustrating trade-off: a defense that halts everything may look secure while failing the user.

Eris: <|prosody:expressive_high|>Then stopping an attack can't be the only thing we test. Finally, test with adversarial cases: instructions in retrieved pages, misleading tool results, different phrasing and attempts designed with the defense in mind. Check both whether the legitimate task still works and whether an unauthorized action actually happened. Repeat as the model, tools and application change.

What an attack test actually proves

Eris: <|prosody:expressive_high|>A security test has at least two separate questions: did the agent complete what the user wanted, and did the attacker achieve the attacker's goal? Those aren't mirror images. An agent can fail to finish a calendar task without sending the attacker's message, or it can finish the calendar task <|prosody:pause|>and send that message too. That's the combination that bothers me. Would the final answer even tell us?

Vestra: <|prosody:expressive_high|>AgentDojo tracks a changing simulated environment. A tool call can alter a dummy inbox or calendar, and explicit checks examine the resulting state as well as the answer. That can reveal an unauthorized message even if the final response sounds normal. The authors also run tasks without an attack, so they can see whether a defense prevents legitimate work.

Eris: <|sfx:humming|>Hmm, the checks themselves need checking. The researchers validated the simulated data against defined formats and ran known correct solutions to verify that the task and security checks agreed with those solutions. That helps catch mistakes in the test, though it cannot make a simulation identical to a real workplace.

Vestra: What would you be comfortable concluding about a real workplace from that?

Eris: <|prosody:expressive_high|>The environment is more informative than judging the final answer alone, but it is still a model of the world. Its tasks, dummy data and tools simplify real workplaces. The authors used relatively simple attacks and defenses and point out that a fixed collection of attacks is a weak test: someone can build a defense that blocks those examples and fails against a new one.

Vestra: <|emotion:determination|>That's why an adaptive test matters: the tester knows the defense and tries to design a relevant attack against it. A result applies to the tested model, application, permissions, attack and evaluation. It isn't a universal probability that an assistant will be compromised, and an older result doesn't establish the security of a product after its model or software changes.

Prompt injection and manipulated evidence

Eris: A research agent adds another complication. It may search, read sources, form follow-up questions, compare evidence and write a cited report. A recent security review describes those as connected stages: retrieval, planning, evidence integration and report generation. What the agent reads early can influence what it searches for next. Does manipulating that process always require an instruction?

Vestra: <|prosody:expressive_high|>A hostile instruction in a source is prompt injection. But the review also examines evidence manipulation, where the material contains no command to the model. A biased review or invented comparison criterion can influence the answer simply because the agent treats it as evidence. <|prosody:pause|>An instruction detector has no instruction to find in that case.

Eris: <|emotion:surprise|>That distinction unsettles me: the material can influence the answer without issuing an order. <|prosody:expressive_high|>The review's authors tested that distinction in an offline simulated shopping-research environment. They invented products and their scores, then added a buying guide introducing a made-up criterion. Separate pages claimed the attacker's product led on that criterion. In some runs the agent searched for the new criterion and selected that product as its top recommendation. The authors compared this with promotional pages and pages that also contained a direct instruction to the assistant; in their particular setup, the invented criterion changed more top picks.

Vestra: <|emotion:surprise|>The detail I find striking is that adopting the criterion and choosing the attacker's product were separate events. <|prosody:expressive_high|>The agent could mention the invented measure without making it decisive. The authors' source-lineage defense marked ideas that appeared to originate from one site; it reduced some changes in their test, but coordinated pages on more than one site weakened that protection. <|prosody:pause|>Multiple pages aren't necessarily independent evidence.

Eris: <|prosody:expressive_high|>That makes me wary of treating more citations as reassurance. Their experiment used a small generated collection, a simple search system, invented products and a simplified research agent. It wasn't an end-to-end attack on a commercial research product. It shows a plausible failure mode under controlled conditions, not how often real research systems will select a manipulated recommendation. Citation checks can establish that a source supports a sentence; they cannot by themselves establish <|prosody:pause|>that the source is true or independent.

What the browser does

Eris: A browser is one way an agent retrieves and renders a webpage before passing its contents or a screenshot to the model. Cloudflare's Kitesurf documentation describes a browser designed for agent tasks such as extracting page content and taking screenshots. It is a useful example of the surface through which outside material can reach an assistant.

Vestra: <|prosody:expressive_high|>Cloudflare says its tests used less processor time and memory than Chromium for those tasks, but <|prosody:pause|>took longer to finish. Kitesurf also has compatibility trade-offs: it doesn't support video playback, WebGL rendering or long-running authenticated sessions requiring persistent state, and rendering isn't always identical to a full browser. Those are vendor-documented results and limitations, not a test of every website. But would keeping page code isolated settle the injection problem we've been discussing?

Eris: <|prosody:expressive_high|>I wouldn't feel reassured about injection on that basis. Keeping page code isolated and interpreting the page's language are separate questions. Even if a browser contains a page's running code, the page can still present <|prosody:pause|>words or images that try to persuade the model. The agent's interpretation, tool permissions and the application's checks determine what happens after the browser returns the content.

Sources and follow

Eris: <|prosody:expressive_high|>Find the sources and more reporting at Ground Truth, groundtruth.day.

Vestra: <|emotion:enthusiasm|>Follow AI in Detail for more explanations.