“Hi, I got charged twice for order #4471. Can you sort it out?”
Under the hood
How an agent actually works, byte by byte
On the first page an agent was a loop with model.infer(context, tools) where the if/else used to be. This page opens up that one line. You'll see the exact JSON that goes over the wire, what comes back, how a “tool call” really happens, and what the model itself reads. At the end is a complete agent you can run: 112 lines of Python.
Read the transcript
Let’s open up that one line: model.infer. It’s a single web request. Your program sends JSON: the rules, the tool descriptions, and every message so far. The API remembers nothing, so all of it goes, every time. Back comes JSON, with a stop reason. “Tool use” means: the model wants lookup_order, for order 4471. Here’s the call, with an ID. Your code runs it, and sends the result back, tagged with that same ID. Next, the model tries to refund $76. Your check refuses: that’s more than the charge. The error goes back as a result, and the model corrects itself: $38. When the stop reason says “end turn”, the loop ends. Five calls. One list. One loop.
-
1 Your harness
Ordinary code you write. It keeps the list of messages, calls the API, checks and runs tools, and decides when to stop. The code →
-
2 The API
One HTTPS endpoint:
POST /v1/messages. JSON in, JSON out. It is stateless: it remembers nothing between calls. The request → -
3 The model
Reads one long sequence of tokens and writes more tokens, one at a time. It knows nothing about JSON, HTTP, or your functions. The tokens →
What's real here. Every request body on this page is exactly what the Anthropic Python SDK sent when we ran the agent below. The responses were scripted in the documented format, so IDs, token counts, wording and the thinking signature are illustrative (the bodies won't replay against the real API). They include one deliberate model mistake so you can see the error path. The first page simplified this run to four calls; the full version here also leaves an internal note. The token view uses a real open-weights model's published template and tokenizer.
What you send: one HTTP request
One inference is one HTTPS POST. Here is the first call of our run, complete. The SDK builds exactly this; you could send it with curl.
model: which model runs the inference.max_tokens: a hard cap on the length of this one response, not the whole run. If it's hit, generation stops mid-sentence andstop_reasonis"max_tokens".system: the system prompt. Explained below.tools: the menu of functions the model may ask for. Explained below.messages: the conversation so far. On call 1 that's just the ticket. Explained below.
Key order doesn't matter in JSON. We reordered keys for reading (the SDK sends them alphabetically) and pretty-printed the body. Nothing else changed.
The system prompt
system is a top-level field: a string, or a list of text blocks if you want to mark cache breakpoints. It is not the first entry of messages; it sits beside it. It holds your instructions as the operator: who the model is, what it's for, what it may do, and when it's finished. Ours has one line per job:
| Line | Job |
|---|---|
You are the support agent for Tidy Home, an online store. | Role and setting |
Resolve the customer's ticket using the tools. Look up an order before acting on it. | Goal and method |
Refunds: only for duplicate or failed charges, and never more than the charge being refunded. | Policy (and your code enforces it too, as call 2 shows) |
When the issue is resolved, send the customer one short, friendly reply, add an internal note, and stop. | Done criteria: when to end the turn |
If you cannot resolve it, escalate to a human. | Fallback |
- A prompt is a request, not a guarantee. Hard limits, like never refunding more than is left on a charge, are also checked in code.
- Keep out: the customer's words (those go in
messages), secrets (the model can repeat anything it reads), and per-tool usage details (those go in each tool'sdescription).
Tool definitions
Each tool is a name, a description, and an input_schema written in JSON Schema. The model never sees your Python function, only this JSON. So the description is prompt text: say when to call the tool, not just what it does (“Call this before acting on any order”).
- Tools cost tokens. Definitions are fed to the model as input, and sending
toolsalso adds a hidden tool-use system prompt: 286 tokens on Claude Opus 5, per Anthropic's pricing docs. You'll see what that looks like. - Want guaranteed-valid arguments? Add
"strict": trueto a tool (its schema also needs"additionalProperties": false) and the model'sinputis guaranteed to match the schema.
The user input and the messages list
messages is an array of {"role", "content"} objects. The rules are few and strict:
- Conversation turns are
"user"and"assistant", alternating, starting withuser. (Consecutive same-role turns get merged into one. Newer models such as Claude Opus 5 also accept an occasional mid-conversation"system"entry, but the top-level field is the normal place.) contentis a string, or a list of typed blocks."content": "Hi…"is shorthand for[{"type": "text", "text": "Hi…"}].- You send
text,image,documentandtool_resultblocks. The model writestext,thinkingandtool_useblocks. - Tool results travel in a
usermessage. From the API's point of view, your harness is the “user” reporting back. - The API is stateless. Every call re-sends the entire list. The agent's “memory” is this array in your program, nothing more.
What comes back: content blocks and a stop reason
The response to call 1. The model has decided it needs data, so instead of answering it asks for a tool. It doesn't run anything; it returns a description of the call it wants.
id,type,role: this is an assistant message. Append itscontenttomessagesunchanged.thinking: Claude Opus 5 decides for itself whether to think before answering. The reasoning text is omitted by default (empty string), but the signed block still comes back. Pass it back untouched.text: ordinary prose the model wrote. It can appear before, between, or instead of tool calls.tool_use: the tool call. It's just data: a uniqueid, the toolname, andinput, already parsed JSON that normally follows your schema (guaranteed only withstrict). Your code decides whether to run it.stop_reason: why the model stopped writing. This is the one field your loop branches on (table below).usage: tokens read and written, which is what you pay for.input_tokenscounts everything re-sent (minus any prompt-cache hits, reported separately), so it grows every call.
Every stop_reason, and what your loop does
stop_reason | Meaning | Your loop |
|---|---|---|
tool_use | The model wants one or more tools run. | Run every tool_use block, send all results back in one user message, call again. |
end_turn | It finished naturally. | break: use the text. |
max_tokens | It hit your max_tokens cap. | Output is cut off, maybe mid tool call. Don't run it; raise the cap or continue. |
stop_sequence | It wrote one of your stop_sequences. | Read stop_sequence to see which, and handle it. |
pause_turn | A server-side tool loop (e.g. web search) paused. | Send the assistant content back to continue. Never happens with only your own tools. |
refusal | Claude declined to respond. | Run nothing. Read stop_details; stop, or retry on a fallback model. |
model_context_window_exceeded | The response filled the context window. | Treat as truncated; shrink the history. |
One run, five calls, every byte
The complete run. For each call you see what the harness appended to messages before sending, and the full response. Watch each tool_use.id come back as a tool_result.tool_use_id, and watch input_tokens climb as the history is re-sent.
All 5 calls shown · messages sent: 9 · input_tokens: 1,823
messages in your program
- user Hi, I got charged twice for order #4471…
- assistant thinking · text ·
tool_use lookup_order - user
tool_result→ order JSON - assistant
tool_use issue_refund(76.00) - user
tool_resultis_error: true - assistant text ·
tool_use issue_refund(38.00) - user
tool_result→ refund succeeded - assistant
tool_use send_reply+tool_use add_internal_note - user 2 ×
tool_resultin one message - program
stop_reason == "end_turn"→ return
Call 1 · the ticket goes in
Appended to messages
Response
The model can't reach your order database. It asks. Your harness runs lookup_order("4471") and gets back the order: the customer's name, a $38 total, and two captured $38 charges, ch_8812 and ch_8813.
Call 2 · the result goes back, and the model makes a mistake
Appended to messages
Response
Two messages were appended: the model's own previous turn, verbatim, then your tool_result, whose tool_use_id matches the request. Now the model asks to refund $76, both charges added up. The system prompt said not to, but a prompt isn't a guarantee. Your run_checked compares the amount with the charge and refuses. Nothing is refunded.
Call 3 · the error goes back as data
Appended to messages
Response
"is_error": true tells the model the call failed, and the content says why. No exception reaches the top of your program; the model reads the error and corrects itself. The refund of $38 passes the check and runs.
Call 4 · two tool calls at once
Appended to messages
Response
Replying and leaving a note don't depend on each other, so the model asks for both in one response: parallel tool calls. Your harness runs both (in any order, or concurrently).
Call 5 · both results in one message, then done
Appended to messages
Response
Both results go back in a single user message, one tool_result per tool_use. (Anthropic warns that splitting them across messages makes the model less likely to call tools in parallel.) This time the response has no tool_use and stop_reason is "end_turn", so the loop breaks.
The tally
- 5 inference calls. Each one was a full HTTP request carrying the whole history.
- 5 tool calls requested, 4 run. One was blocked by your code, and the block became information the model used.
input_tokenswent 1,165 → 1,823. You pay for the history every call, which is why long agent runs get expensive and why prompt caching exists.- The model never touched your systems. It wrote JSON; your harness decided what ran.
What the model actually reads: tokens
Read the transcript
Here’s the twist… the model never sees your JSON. Before inference, the server flattens everything (system prompt, tools, messages) into one long piece of text, using a chat template. Little markers say who’s talking: system, user, assistant. Then that text is chopped into tokens: little numbered pieces of words. That’s all the model ever reads… a long row of numbers. And all it does is predict the next token. Then the next. Then the next. When the tokens it writes spell out a tool call, a parser turns them back into JSON, and hands it to your program. So a “tool call” is just text the model wrote. Your code decides what it means. (The markers and token ids shown are Qwen3-8B’s, for example; Claude’s own format isn’t published.)
The model never sees your request as a JSON object. Before inference, the request is flattened: the server turns system, tools and messages into one long string using a chat template, splits it into tokens (integers), and the model predicts the next token, over and over.
Why a different model here? Claude's internal format isn't published. So this is the conversation as of call 3, rendered by the published template of an open-weights model, Qwen3-8B, and split by its real tokenizer. The markers differ between models; the mechanism is the same.
1 · The JSON you send (call 3)
2 · The same conversation after the chat template. Special tokens highlighted.
<|im_start|>system You are the support agent for Tidy Home, an online store. Resolve the customer's ticket using the tools. Look up an order before acting on it. Refunds: only for duplicate or failed charges, and never more than the charge being refunded. When the issue is resolved, send the customer one short, friendly reply, add an internal note, and stop. If you cannot resolve it, escalate to a human. # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within <tools></tools> XML tags: <tools> {"type": "function", "function": {"name": "lookup_order", "description": "Get an order's total and every charge made for it. Call this before acting on any order.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string", "description": "Digits only, e.g. 4471"}}, "required": ["order_id"]}}} {"type": "function", "function": {"name": "issue_refund", "description": "Refund one charge, fully or partly. Amount must not exceed that charge.", "parameters": {"type": "object", "properties": {"charge_id": {"type": "string"}, "amount": {"type": "number"}}, "required": ["charge_id", "amount"]}}} {"type": "function", "function": {"name": "send_reply", "description": "Email the customer. Use once, when the ticket is resolved.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}} {"type": "function", "function": {"name": "add_internal_note", "description": "Leave a note on the ticket for the support team. The customer never sees it.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}} {"type": "function", "function": {"name": "escalate_to_human", "description": "Hand the ticket to a person. Use when the tools cannot resolve it.", "parameters": {"type": "object", "properties": {"reason": {"type": "string"}}, "required": ["reason"]}}} </tools> For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> {"name": <function-name>, "arguments": <args-json-object>} </tool_call><|im_end|> <|im_start|>user Hi, I got charged twice for order #4471. Can you sort it out?<|im_end|> <|im_start|>assistant I'll pull up order #4471 first. <tool_call> {"name": "lookup_order", "arguments": {"order_id": "4471"}} </tool_call><|im_end|> <|im_start|>user <tool_response> {"order_id": "4471", "customer": "Priya Shah", "total": 38.0, "charges": [{"id": "ch_8812", "amount": 38.0, "status": "captured"}, {"id": "ch_8813", "amount": 38.0, "status": "captured"}]} </tool_response><|im_end|> <|im_start|>assistant <tool_call> {"name": "issue_refund", "arguments": {"charge_id": "ch_8813", "amount": 76.0}} </tool_call><|im_end|> <|im_start|>user <tool_response> Refund of 76.00 rejected: charge ch_8813 has 38.00 left to refund. Nothing was refunded. </tool_response><|im_end|> <|im_start|>assistant <think> </think>
3 · 780 tokens. The model receives only the integers: 151644, 8948, 198, 2610, 525, 279, 1824, 8315, 369, 350, 42586, 5379, … On a desktop, hover a token to see its id.
<|im_start|>system↵ You are the support agent for Tidy Home, an online store.↵ Resolve the customer's ticket using the tools. Look up an order before acting on it.↵ Refunds: only for duplicate or failed charges, and never more than the charge being refunded.↵ When the issue is resolved, send the customer one short, friendly reply, add an internal note, and stop.↵ If you cannot resolve it, escalate to a human.↵ ↵ # Tools↵↵ You may call one or more functions to assist with the user query.↵ ↵ You are provided with function signatures within <tools></tools> XML tags:↵ <tools>↵ {"type": "function", "function": {"name": "lookup_order", "description": "Get an order's total and every charge made for it. Call this before acting on any order.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string", "description": "Digits only, e.g. 4471"}}, "required": ["order_id"]}}}↵ {"type": "function", "function": {"name": "issue_refund", "description": "Refund one charge, fully or partly. Amount must not exceed that charge.", "parameters": {"type": "object", "properties": {"charge_id": {"type": "string"}, "amount": {"type": "number"}}, "required": ["charge_id", "amount"]}}}↵ {"type": "function", "function": {"name": "send_reply", "description": "Email the customer. Use once, when the ticket is resolved.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}↵ {"type": "function", "function": {"name": "add_internal_note", "description": "Leave a note on the ticket for the support team. The customer never sees it.", "parameters": {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]}}}↵ {"type": "function", "function": {"name": "escalate_to_human", "description": "Hand the ticket to a person. Use when the tools cannot resolve it.", "parameters": {"type": "object", "properties": {"reason": {"type": "string"}}, "required": ["reason"]}}}↵ </tools>↵ ↵ For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:↵ <tool_call>↵ {"name": <function-name>, "arguments": <args-json-object>}↵ </tool_call><|im_end|>↵ <|im_start|>user↵ Hi, I got charged twice for order #4471. Can you sort it out?<|im_end|>↵ <|im_start|>assistant↵ I'll pull up order #4471 first.↵ <tool_call>↵ {"name": "lookup_order", "arguments": {"order_id": "4471"}}↵ </tool_call><|im_end|>↵ <|im_start|>user↵ <tool_response>↵ {"order_id": "4471", "customer": "Priya Shah", "total": 38.0, "charges": [{"id": "ch_8812", "amount": 38.0, "status": "captured"}, {"id": "ch_8813", "amount": 38.0, "status": "captured"}]}↵ </tool_response><|im_end|>↵ <|im_start|>assistant↵ <tool_call>↵ {"name": "issue_refund", "arguments": {"charge_id": "ch_8813", "amount": 76.0}}↵ </tool_call><|im_end|>↵ <|im_start|>user↵ <tool_response>↵ Refund of 76.00 rejected: charge ch_8813 has 38.00 left to refund. Nothing was refunded.↵ </tool_response><|im_end|>↵ <|im_start|>assistant↵ <think>↵↵ </think>↵↵
What to notice
- Roles are just tokens. Each turn is wrapped in
<|im_start|>role…<|im_end|>. The model was trained on huge numbers of conversations in exactly this shape, which is why “system” text carries weight and why APIs keep turns in a strict user/assistant rhythm. - Tools are just text. Your tool list is pasted into the system turn as JSON lines, with instructions on how to call them. That's the equivalent of Claude's hidden 286-token tool-use prompt. Note that
<tools>here isn't even a special token, just ordinary characters. - A tool call is text the model wrote. Earlier assistant turns contain
<tool_call>{"name": …, "arguments": …}</tool_call>. The API'stool_useblocks are this, parsed. - Results come back as a user turn wrapped in
<tool_response>. There's nois_errorfield at this level: the error is just words the model reads. - The prompt ends with an open slot:
<|im_start|>assistant. (The empty<think></think>is there because thinking was switched off for this render.) Whatever the model writes next is its turn.
Generation: one token at a time
Here is call 3's (scripted) answer, split into this model's real tokens. Qwen didn't generate it; the demo shows how any model emits text. Each step, the model scores every token in its vocabulary (about 150,000), one is picked, it's appended, and the model runs again on the longer sequence.
- Only7308
- the279
- duplicate22513
- charge6757
- should1265
- be387
- refunded93452
- ,11
- so773
- the279
- amount3311
- is374
- ␣220
- 318
- 823
- .13
- 015
- 015
- .↵624
- <tool_call>151657
- ↵198
- {"4913
- name606
- ":788
- "330
- issue11159
- _ref7793
- und1241
- ",497
- "330
- arguments16370
- ":788
- {"5212
- charge13891
- _id842
- ":788
- "330
- ch331
- _62
- 823
- 823
- 116
- 318
- ",497
- "330
- amount6045
- ":788
- ␣220
- 318
- 823
- .13
- 015
- }}↵11248
- </tool_call>151658
- <|im_end|>151645
Then the server parses the text back into JSON
<tool_call>is one token (id 151657), added to the vocabulary for this job, so the start of a call is unambiguous to both model and parser.- Numbers are written digit by digit. “38.00” is five tokens. The model doesn't calculate a refund like a calculator; it writes plausible digits. That's one more reason for
run_checked. <|im_end|>(id 151645) is the stop token. When it's picked, generation ends. Ifmax_tokensruns out first, you get half a turn, maybe half a tool call. That's why you checkstop_reasonbefore running anything.- Malformed JSON is just bad text. The parser fails on it. Strict tool use (
"strict": true) prevents this by constraining sampling so only tokens that keep the output schema-valid can be picked.
The same ideas in Claude's API
| In the token stream (Qwen3) | In Claude's API |
|---|---|
<|im_start|>user … <|im_end|> | {"role": "user", "content": …} |
| Tool JSON pasted into the system turn | tools, plus a hidden tool-use system prompt (286 tokens on Opus 5) |
<think> … </think> | thinking block |
<tool_call> … </tool_call> | tool_use block with an id |
<tool_response> … </tool_response> | tool_result block with tool_use_id and optional is_error |
| End-of-turn token | stop_reason: "tool_use" or "end_turn" |
The whole agent, in one file
Everything above in one runnable file. Run pip install anthropic, set ANTHROPIC_API_KEY, then python agent.py. The orders are fake, so it's safe to run; swap the function bodies for calls to your real systems.
Mapped to the first page's pseudocode
| First page | This file |
|---|---|
context = [SYSTEM_PROMPT, ticket.body] | system=SYSTEM_PROMPT and messages = [{"role": "user", "content": ticket}] |
move = model.infer(context, tools) | response = client.messages.create(…) |
if move.done: break | if response.stop_reason == "end_turn": return … |
check(move) | run_checked(block.name, block.input) |
context.append(result) | messages.append({"role": "user", "content": results}) |
for step in range(MAX_STEPS) | Same, then escalate_to_human if it runs out |
We wrote the loop by hand because showing it is the point. In production you can let the SDK's Tool Runner (client.beta.messages.tool_runner) run this loop over your functions, or use the Claude Agent SDK for a full harness with built-in file, shell and web tools.
What production harnesses add
The loop above is the whole idea. Real harnesses wrap it with engineering, none of which changes the mechanism:
- Streaming
- Show tokens as they're generated instead of waiting for the full response. Needed for long outputs and large
max_tokens. - Retries
- Rate limits and server errors happen. The SDK retries 429 and 5xx responses for you (twice by default).
- Context management
- Re-sending everything gets expensive. Prompt caching reuses the unchanged prefix; compaction summarizes old turns.
- Approval gates
- Pause before side-effecting tools, like refunds, and ask a person. It's the same spot as
run_checked. - Budgets
- Step caps, token budgets, timeouts, and a spending limit per run.
- Fallbacks and sub-agents
- Retry a declined request on another model; let one agent be a tool of another. It's the same loop, nested.
An agent is a list of messages, one POST request, and a loop. The model writes text; a parser turns some of that text into tool calls; your code decides what runs and when to stop.
← Back to the idea Play Loop Lab
Primary sources
- Anthropic: Tool use with Claude (round trip, and the tool-use system prompt token counts)
- Anthropic: Handle tool calls (tool_result format, is_error, parallel results)
- Anthropic: Handling stop reasons (every stop_reason value)
- Anthropic: Strict tool use (schema-guaranteed tool inputs)
- Anthropic: Messages API reference (every request and response field)
- Qwen3-8B chat template and tokenizer (the template and tokenizer used for the token view)