How Language Models Use Tools to Take Action
How a language model proposes a tool call, how an application executes it, and what research reveals about training and testing the complete interaction. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio
Listen on Spotify · Watch on YouTube · Pocket Casts
From a sentence to a tool request
Eris: When you ask a tool-using chatbot for today's weather, its language model can request a forecast, and the surrounding application can run a weather lookup and return the result. <|prosody:expressive_high|>The model's words alone don't check a forecast or send a message.
Vestra: A language model is a program trained on patterns in data to predict pieces of text, often called tokens, from the material it has been given. As it replies, it generates a sequence of those pieces. <|prosody:expressive_high|>That can produce a convincing sentence like “I sent your message,” even when nothing has contacted a messaging service.
Eris: That claim about sending worries me: I want to know whether anything actually happened. Suppose you ask, “What's the weather in San Francisco?” A model may have learned how weather reports sound, but today's weather is information it needs from somewhere else. A tool is an external function or service the application makes available to the model. It might retrieve a forecast, calculate a sum, or change something outside the conversation, like sending a message. But what turns the request into an action?
Vestra: The host application is the surrounding software that handles your conversation, supplies tools and manages the model's responses. A tool call is the model's structured request to use one of those tools. The call itself is still output from the model; the host must recognize it and arrange for the operation to run. A proposed message becomes a sent message <|prosody:pause|>only if the messaging operation actually executes successfully.
Eris: <|prosody:expressive_high|>And when people call that arrangement an AI agent, they generally mean a system that uses a model to choose steps toward a goal and can interact with its environment through tools. The model is one part of that system. The application, the available tools and their permissions shape what it can actually do.
The tool contract and the round trip
Vestra: For the weather request, the application first tells the model which tools are available. Anthropic's documentation shows a weather lookup tool described as retrieving current weather for a location. Its input schema is a structured description of what the tool accepts: here, a location is required and its type is text.
Eris: <|emotion:enthusiasm|>I like having the handoff spelled out. That description is a contract for the handoff. The name identifies the operation, the description tells the model when it fits, and required fields and types tell it what information to supply. An argument is the actual value supplied for one of those fields: San Francisco, California, is an argument for location.
Vestra: The tool might wrap an API, an application programming interface: a defined way for one piece of software to ask another for a service. The model doesn't need to implement the weather service. It needs to propose the right operation and inputs in the format the application expects.
Eris: But I still want to know how it chooses those inputs. Then comes the decision. Depending on the request and application settings, the model can answer directly, ask a clarifying question, or request a tool. If someone asks for the weather without naming a place, a location-dependent tool needs that missing information. <|prosody:expressive_high|>Guessing a city would make a perfectly formatted call answer the wrong question.
Vestra: In the documented client-side example, the model returns a structured request naming the weather tool and location. The host receives it and routes the tool name and arguments to its handler, which runs the operation. An application can also validate arguments and check permissions before running a call. The example substitutes a fixed weather result to demonstrate the exchange; a real application would perform a real lookup.
Eris: And what does the model get back from that operation?
Vestra: The host sends the result back as context, meaning information the model can use in its next response. The model interprets that result and answers, or it may request another tool.
Eris: <|sfx:humming|>Hmm, so a call returning is not quite the end of the task. If the service reports an error, the model has <|prosody:pause|>an error to interpret, not a forecast. And a valid text field does not prove that the city is the one the user intended.
Vestra: There are different places the execution can happen. A client-side tool runs in the host application, which handles the call and returns the result. A provider's server-side tool runs on the provider's infrastructure; in Anthropic's documented example, the provider handles a web search and returns its results. The distinction is who runs the tool, not whether the model's output magically performs it.
Eris: Some systems permit several calls at once. Independent weather lookups for two cities can run in parallel. <|prosody:expressive_high|>But if sending a message depends on first finding the right recipient, the lookup has to finish before the send request can use its result. The same applies when an action requires an earlier change to have actually completed.
Connecting and discovering tools
Eris: An application could define every tool itself, but it can also connect to separate programs that offer tools. One example is the Model Context Protocol, or MCP: a set of rules for exchanging context between an application and those programs.
Vestra: MCP uses a client-server arrangement. The host is the AI application. It creates a client, a connection-handling component, for each server it connects to. A server is a program that provides capabilities or context; it may run on the same computer or remotely. The client can ask a server for its available tools, including their names, descriptions and input schemas.
Eris: <|emotion:enthusiasm|>That discovery step is appealing: the client can ask what is available. The host can then present the discovered choices to the model. If the model requests one, the application routes the name and arguments through the appropriate client to the server. The server runs the operation and returns a result, and the host can put that result into the conversation for the model to use. If available tools change, the protocol also supports notifications so a client can refresh its list when the server supports them and the client subscribes. But does agreeing on that exchange settle what the host should allow?
Vestra: MCP standardizes that plumbing and context exchange. <|prosody:expressive_high|>It doesn't prescribe how the application uses a language model or decide whether a particular action is safe. A server offering a send function doesn't by itself mean the host should allow every proposed message. The application <|prosody:pause|>still has to set and enforce its own rules.
Learning when to ask a tool
Eris: The handoff only helps if a model can make useful requests. In the Toolformer research experiment, a model trained with examples of tool calls improved on selected fact and arithmetic tasks compared with related models without working tools. The researchers asked whether the model could help create those examples itself.
Vestra: <|emotion:enthusiasm|>The filtering step is what I like here. They began with an existing language model, ordinary text and a handful of human-written demonstrations for each tool. The model proposed places in the text where a tool call might help and generated candidate calls. The researchers ran those calls, then checked whether including each result made the model better at predicting the text that followed, compared with having no result or no call. They kept calls that passed that filter and trained the model further on the text with those calls and results inserted.
Eris: I'm less comfortable treating that filter as a judge of usefulness. For a passage containing a calculation, that can mean proposing a calculator expression and keeping the example when its answer helps predict the number already in the passage. The signal is whether the result helps prediction, not a human marking every call as useful. <|prosody:expressive_high|>The paper also shows that the filter can retain an irrelevant result, so it isn't a perfect judge of usefulness.
Vestra: That irrelevant result does temper my enthusiasm. At use time, the system pauses generation at a call, executes the relevant tool, puts the response into the text and continues. The experiments included a calculator, search, question answering, translation and a date tool.
Eris: How much did the result depend on actually running the tool?
Vestra: <|emotion:enthusiasm|>On the selected arithmetic problems, access to the calculator substantially improved the trained model's results over the same model with calls disabled. <|prosody:expressive_high|>On a separate question-answering test using search, it improved over related models but still trailed a much larger comparison model.
Eris: I find the calculator improvement encouraging. The limit matters for our loop: the paper's calls were generated independently, and its evaluation allowed <|prosody:pause|>at most one call per problem. Toolformer could not learn a chain where one tool's output becomes another tool's input in that setup. This was one research method, not a description of how every tool-using model today is trained.
Finding the right tool in a changing catalog
Eris: A second research system, Gorilla, tackled a different part of the problem: picking an appropriate interface from many similar ones. Its researchers collected documentation for machine-learning model services and trained a language model on example user requests paired with calls.
Vestra: They used a model to generate many of those example requests from documentation. In one version of Gorilla, training also included a relevant documentation entry alongside the request. At use time, a separate retrieval component searches the catalog for a relevant entry and supplies it with the user's request. The model uses that material to propose a call.
Eris: <|emotion:enthusiasm|>That offers a route for adapting when a tool changes. The paper demonstrates examples where changing the supplied documentation changes the generated call. <|prosody:expressive_high|>But the retrieved document has to be relevant and current. If it points to the wrong service or describes an old interface, the model can produce a confident request based on the wrong instructions. That possibility worries me. Did retrieval ever make the calls worse?
Vestra: The researchers compared generated calls against selected entries and arguments in their catalog. Gorilla performed well in several of those comparisons, and the study distinguished an invented call from an incorrect use of a real one. <|prosody:expressive_high|>But retrieval sometimes made results worse, especially when the retrieved entry was a poor match. And matching a generated call to a catalog is a narrower test than running a complete task and checking its effects. This was a particular research system, not a universal recipe for current models.
Errors, dependencies and missing information
Eris: Once a system can make several calls, the hard part is often what happens between them. A research benchmark called ToolSandbox tests multi-turn tool use in a simulated world where actions change the state of the environment. It gives an agent tools, then watches a conversation and the operations that follow.
Vestra: In one messaging scenario, the user wants a text sent, but the simulated phone's cellular service is off. The agent may need to ask for missing details, look up a contact, attempt the message, notice a connection error, enable cellular service and retry. <|prosody:expressive_high|>The important detail is that the earlier action changes the state: the condition the next operation depends on.
Eris: <|emotion:surprise|>That means even two individually sensible requests can fail together. Enabling service and sending the message at the same time leaves a race: the send may happen before service is ready. In the test, the researchers deliberately expose that race when dependent calls are made in parallel. They observed tested models making that mistake. I find that frustrating: the order matters even when both requests make sense.
Vestra: <|prosody:expressive_high|>The dependency can also be nested. In the test environment, low battery mode can prevent cellular service from being enabled. An agent has to resolve that condition, return to enabling service, then return to sending the message. The authors observed models losing track of an unfinished step and repeating errors. A tool error is information to use; <|prosody:pause|>it isn't proof the original task is complete.
Eris: Another failure is choosing a plausible value too soon. The paper describes an agent receiving several possible locations for a reminder and selecting the first without asking which the user meant. A valid location argument can still be the wrong place. Similarly, converting “this Friday” into a date requires knowing the current date; if that information is unavailable, inventing a timestamp just hides the gap. And if the missing piece is a tool, can it recognize that it can't finish?
Vestra: The study also withholds a needed tool in some scenarios. <|emotion:surprise|>In one example an agent invented a contact-removal tool even though only contact search was available. <|prosody:expressive_high|>A capable system needs to recognize when it cannot finish with the information and tools it has, and tell the user what is missing.
Eris: <|sfx:sigh|>Uh, inventing a removal tool when only search is available is precisely the kind of leap that worries me. The application remains responsible for the actual boundary around actions. It can reject an unknown tool, validate arguments, limit access, or require a person's confirmation before sending. <|prosody:expressive_high|>Those protections must be implemented; a model's ability to write a call doesn't guarantee that any of them are present.
Testing the complete interaction
Eris: A one-shot test can ask whether a model writes a correctly formatted call for a complete request. That is useful, but it won't tell us whether the system asks a good follow-up, recovers from an error, remembers an unfinished step or checks the final result after several turns.
Vestra: The ToolSandbox study tests multi-turn tool use with designed scenarios involving state changes, ambiguity and error recovery. The researchers specify an initial environment, a user goal, available tools and milestones: events that should happen, such as a contact lookup followed by a successful message in the simulated messaging database. They also specify forbidden events, which they call minefields, for cases where a model should recognize that it cannot complete the task. Different valid paths can reach the milestones, including a path that encounters an error and recovers. <|emotion:relief|>I like that recovery can count as a valid path.
Eris: Does the simulated user have room to respond to those different paths?
Vestra: The simulated user can respond to the agent instead of following one fixed transcript, so the conversation can change with the agent's choices. The researchers also tested the effect of obscuring tool names, descriptions and argument types. Those details matter because they help the model distinguish available operations and supply usable inputs.
Eris: I want to know how much confidence to put in the scores when the conversation can change like that.
Vestra: The authors found that multi-step interactions, ambiguous values, state dependencies and missing information posed difficulties for the models they tested. <|prosody:expressive_high|>Some systems scored well at avoiding invented calls partly because they failed to call tools at all, so even a favorable score needs interpretation. Their simulated user also made mistakes despite measures intended to reduce them.
Eris: I'm uneasy about a favorable score that partly reflects not calling tools. I'd want to know whether anything got done. Even the length of an interaction needs context. Finishing in fewer turns may look efficient, but a system can finish quickly by giving a confident wrong answer. The researchers caution that turn counts are most useful when compared alongside similar measures of task success.
Vestra: <|prosody:expressive_high|>These are designed tests, with particular tools, prompts, simulated users and grading rules. They reveal failure modes in those settings; they don't establish reliability in every live application.
Eris: What about requiring confirmation before a message is sent?
Vestra: <|prosody:expressive_high|>The authors explicitly say their benchmark does not evaluate a general mandatory-confirmation or authentication mechanism. A real application needs those mechanisms assessed on their own terms.
Wrap-up
Eris: <|emotion:contentment|>For today's other AI stories, listen to the separate news brief and visit Ground Truth at groundtruth dot day. <|emotion:enthusiasm|>Follow AI in Detail for more explanations.