Tool calling is structured output with consequences

The first time you watch a model "call a function", it looks like the model reached out and did something. It didn't. The model wrote a small block of JSON that says which function it wants and with which arguments. Your code read that JSON and decided whether to run anything.

That distinction is the whole game. Tool calling is structured output that has consequences, and nearly every reliability problem I've hit with agents comes down to how well the tools were designed and how carefully the harness handled what the model wrote.

What actually happens

A tool-enabled request has three parts: a system prompt, a list of tool definitions and the conversation so far. Each tool definition is a name, a description and a JSON Schema for its arguments. The model reads all of it as text. It has no special access to your functions. It only knows what the schema and description tell it.

{
  "name": "find_customer",
  "description": "Search customer master records by legal name. Returns up to 'limit' candidates with id, name, country and tax_id. Call this before any tool that needs a customer_id.",
  "input_schema": {
    "type": "object",
    "properties": {
      "name":    { "type": "string", "description": "Full or partial legal name, e.g. 'Acme Corp'" },
      "country": { "type": "string", "enum": ["US", "DE", "IN", "GB"], "description": "ISO code. Omit to search all countries." },
      "limit":   { "type": "integer", "minimum": 1, "maximum": 20, "default": 5 }
    },
    "required": ["name"]
  }
}

When the model decides a tool would help, it replies with a tool_use block (a function call, in OpenAI's terms) containing an ID, the tool name and the arguments as JSON. It also sets a stop reason that says "I'm waiting on a tool". Then it stops generating. From here on, it's your turn.

system: "You are a data steward…"
tools: [
  find_customer
  match_score
  merge_customers
]
messages: [user]

stop_reason: "tool_use"
content: [
  tool_use {
    id: "toolu_01"
    name: "find_customer"
    input: { name: "Acme Corp" }
  }
]

The harness validates the arguments, runs the function and appends a tool_result message that references the call's ID. With Anthropic's API, that result goes inside a user message; with OpenAI's, it's a message with the tool role. Then you call the model again with the longer conversation. The model reads the result as text and decides what to do next: call another tool or answer.

user        Are "Acme Corp" and "ACME Corporation GmbH" the same customer?

assistant   tool_use     toolu_01  find_customer {"name": "Acme Corp"}
            tool_use     toolu_02  find_customer {"name": "ACME Corporation GmbH"}
            stop_reason: tool_use

user        tool_result  toolu_01  [{"id": "C-10482", "country": "DE", "tax_id": "DE811…07"}]
            tool_result  toolu_02  [{"id": "C-20931", "country": "DE", "tax_id": "DE811…07"}]

assistant   Very likely the same company. Both records share tax ID DE811…07 ...
            stop_reason: end_turn

Parallel calls

Notice the assistant asked for two lookups in a single turn. Modern models can emit several independent tool calls at once, and the harness can run them concurrently and return all the results together. For lookups like these, that halves the number of round trips.

The catch is that the model decides what's independent. Two reads are fine in parallel. A read followed by a write that depends on it is not. If your tools have ordering constraints, enforce them in the harness, or turn parallel calls off for that agent.

Tool design matters more than prompt design

When an agent picks the wrong tool or passes bad arguments, my first instinct used to be adding a line to the system prompt. Now I look at the tool first. The model sees the tool definitions on every call, and a precise schema does more than a paragraph of instructions ever will.

Fewer, sharper tools

Every tool you add is one more option the model has to weigh on every step. I aim for the smallest set that covers the task, where each tool has one clear job. If two tools overlap, the model will sometimes choose the wrong one, and you'll spend days figuring out why.

Names and descriptions are prompts

find_customer tells the model what the tool does. crm_query_v2 doesn't. The description should say what the tool returns, when to use it and when not to. "Call this before any tool that needs a customer_id" prevents a whole class of made-up IDs.

Types over prose

If an argument has five valid values, make it an enum. If it's a number with limits, put the limits in the schema. Many providers now offer strict modes that constrain generation to your schema, which removes most malformed calls outright. You still validate on your side, but you'll catch far fewer problems.

Return what the model needs, not what the API returns

Tool results land in the context window and stay there. A tool that returns a full API response pushes out the information the model actually needs, and it gets billed on every later turn. I shape results down to the fields that matter, cap list lengths and paginate. If the model needs more, it can ask.

Errors the model can act on

"HTTP 500" leaves the model with nothing to do except retry. "No customer named 'Acme Corp' in DE. Two matches in AT. Retry with country='AT' or search by tax_id" gives it a next step. Write errors for the model as the reader, the same way you'd write them for a junior engineer.

Controlling when tools get called

By default the model decides whether to call a tool at all. Most APIs let you override that with a tool_choice setting: let the model decide, require some tool, require one specific tool or allow none. I use a forced single tool mainly for extraction steps, where the "tool" is really a schema for the output I want. Classifying a record or pulling fields from a document works this way, and the model can't reply in free text instead.

The other lever is which tools you expose at each step. An agent in a triage phase doesn't need the merge tool in its list. Dropping tools that don't apply to the current phase shrinks the prompt and rules out a whole category of mistakes before the model has a chance to make them.

The consequences part

A tool that reads data and a tool that merges customer records can look the same to the model. They shouldn't look the same to your harness. I tag every tool with a side-effect tier and attach a policy to each tier.

A few habits make the write tiers safe:

• Validate twice. Check the JSON against the schema first, then apply business rules. The schema can confirm that customer_id is a string. Only your code can confirm the customer exists and the user is allowed to touch it.
• Make writes idempotent. Models retry, networks drop and harnesses resume from checkpoints. Derive an idempotency key from the call so the same request can't create two tasks or send two emails.
• Never trust IDs you didn't hand out. If the model passes a record ID, confirm it came from an earlier tool result in this run. A made-up ID that happens to match a real record is the worst kind of bug.
• Gate the irreversible. Anything that can't be undone waits for a person. I cover how that fits into the loop in the harness is the product.

How it goes wrong

These are the failure modes I see most, and the fix that usually works:

1. Invented arguments. The model fills in an ID or email it never saw. Fix: require a lookup tool first, say so in the description, and verify IDs against earlier results.
2. The wrong tool. Two tools with overlapping descriptions. Fix: merge them, or make their descriptions say explicitly when to use each one.
3. Retry loops. The same failing call over and over. Fix: actionable error messages plus loop detection in the harness.
4. Context flooding. One oversized result pushes out everything else. Fix: shape and cap outputs, and offload large payloads to a file or store the model can query.
5. Ignoring results. The model answers from its own assumptions and not from what the tool returned. Fix: keep results short and structured, and put the key fact first.

Close

Tool calling is simple mechanics: a schema in the request, JSON coming back and a result going in. The engineering is in everything around that exchange. Design tools like an API for a capable colleague who can't ask follow-up questions, and let the harness enforce the rules the model can't be trusted to remember.