This page needs no coding background.

Plain English · Page 3 of 4 · No coding background needed

What really gets sent, every time

On the last page, the program “asked the model”. This page opens that up. What exactly gets sent? What comes back? And why does it get more expensive the longer it runs?

It doesn’t remember you: each time, the app sends the whole conversation again.
Read the transcript

Ever had a chatbot remember something you said earlier? Here’s the trick… the model didn’t. The AI model, the part that writes replies, keeps nothing between messages. Each time, the app sends everything again: its instructions, the whole conversation so far, and your new message. That bundle is called the context. Priya’s first message: one short note. Her second: that note, the reply, and the new one. By the tenth, it’s a long scroll. The model reads the whole thing, from the top, every single time. That’s its memory: a longer note, sent again. Reading isn’t free. Every token, every little piece of text, is counted and paid for, so long chats get slower and pricier. And there’s a limit: past it, something gets cut, or summed up. So when it “remembers” your name… your name is still on the scroll. No memory. A longer note, every time.

There are three players:

  1. 1 The program

    Ordinary software the store wrote. It keeps the note, sends it, checks every action request, carries out the allowed ones, and decides when to stop.

  2. 2 The model service

    Computers run by an AI company, such as OpenAI, Anthropic or Google, that run the model and send back its reply, reached over the internet. You send a package; you get a reply. It keeps no record of your conversation between packages.

  3. 3 The model

    The formula from page 1. It reads one long row of tokens and writes more tokens, one at a time. It has no connection to the store’s refund system.

Because the service keeps no record of your conversation, the program must send everything, every time.

That is also all that memory means in AI apps: notes that the app saves and re-sends. The model itself keeps nothing between requests.

Running example · the same request as on “What agents are”

“Hi, I got charged twice for order #4471. Can you sort it out?”

What’s real here. The instructions and action menu below are the real text from the worked example on the With-code page. The model’s replies in that example were written by us, not by a live model, so the token counts are illustrative. The flattened text near the end, and its token count, come from a real open model (Qwen3).

See this with code

What goes in the package

Every trip, the program sends one package with three parts.

  1. 1 Standing instructions written by the store, same every trip
  2. 2 Action menu 5 actions, same every trip
  3. 3 The conversation so far grows on every trip
One package, three envelopes. All of it goes on every trip.

1. The standing instructions

The standing instructions are directions written by whoever built the app, sent at the top of every request. People in the field call this the system prompt. Here is the real text, line by line:

The standing instructions, line by line, with what each line is for
LineWhat it’s for
You are the support agent for Tidy Home, an online store.Who and where
Resolve the customer’s ticket using the tools. Look up an order before acting on it.The goal, and how
Refunds: only for duplicate or failed charges, and never more than the charge being refunded.The policy
When the issue is resolved, send the customer one short, friendly reply, add an internal note, and stop.When it’s finished
If you cannot resolve it, escalate to a human.What to do if stuck

“Tools” is the industry word for actions. A “ticket” is one customer request. “Escalate” means hand to a person. In the menu below, “call” means “use”.

2. The action menu

The action menu is the list of allowed actions, each with a short description, sent to the model every time. The model never sees how an action works, only this description. So the description has to say when to use it.

3. The conversation so far

On the first trip, that’s Priya’s message. On later trips it’s her message, plus every action request the model wrote, plus every result the program wrote back.

All three parts together are the prompt: everything sent to the model in one go. You’ll also hear it called the context: everything the model is shown on one request, the whole note it reads before predicting.

The menu and instructions aren’t free. They’re text, so they’re counted in tokens: small pieces of text, such as a word, part of a word, a space plus a word, a single digit or a punctuation mark. And they’re re-sent on every trip.

The instructions are a request, not a guarantee. The “never more than the charge” line will matter in a minute.

See this with code

What comes back

The model service sends back a reply with up to four things in it:

  • Text the model wrote, like “I’ll pull up order #4471 first.”
  • An action request, if the model asked for one: text that names an action and its details, like “Look up an order · order number 4471”. It gets a reference number, so the program can say which request a result belongs to.
  • A label saying why it stopped writing. The two everyday ones: “action requested” or “finished”. (There’s also “ran out of room”, when the reply hit a length cap. Then the program shouldn’t trust a half-written request.)
  • A count of tokens read and tokens written. The bill is worked out from these.

Reply · trip 1

“I’ll pull up order #4471 first.”

ref #1 Look up an order · 4471

  • action requested (ticked)
  • finished (not ticked)

read 1,165 · wrote 96 tokens

The reply to trip 1, drawn as a slip: some text, one action request with its reference number, the label, and the count.

Nothing has happened yet.

The model hasn’t looked anything up. The reply is text asking for something to be done. What happens next is up to the program.

See this with code

One message, five trips

Here’s the whole job: five trips. Watch the package get thicker, and watch the “tokens read” counter climb, because every trip re-sends everything before it.

All 5 trips shown · the final package holds 9 entries · tokens read on trip 5: 1,823

The package

Always on top: the standing instructions and the action menu

  1. Priya Hi, I got charged twice for order #4471. Can you sort it out?
  2. model “I’ll pull up order #4471 first.” Request (ref #1): look up order 4471
  3. program Result for ref #1: order 4471, total $38, two charges of $38
  4. model Request (ref #2): refund $76 on the second charge
  5. program Refund of $76 rejected: that charge has $38 left. Nothing was refunded.
  6. model “Only the duplicate charge should be refunded, so the amount is 38.00.” Request (ref #3): refund $38
  7. program Refund of $38 succeeded.
  8. model Two requests: email Priya (ref #4), leave a team note (ref #5)
  9. program Email sent. Note saved.

Trip 1 · Priya’s message goes in

Added to the package
Priya’s message
The reply
Action request: look up order 4471
Label
action requested
Tokens read · written
1,165 · 96

The model can’t reach the store’s records, so it asks. The program looks the order up.

Trip 2 · The order comes back, and the model asks for too much

Added to the package
That request + the order details (two $38 charges)
The reply
Action request: refund $76 on the second charge
Label
action requested
Tokens read · written
1,341 · 71

It asks for $76: both charges added together. The instructions said not to. The program’s check compares $76 with the $38 charge and refuses. Nothing is refunded.

Trip 3 · The refusal goes back as words

Added to the package
That request + “Refund of $76 rejected: that charge has $38 left. Nothing was refunded.”
The reply
“Only the duplicate charge should be refunded, so the amount is 38.00.” + action request: refund $38
Label
action requested
Tokens read · written
1,468 · 88

The refusal went back as plain words in the package. With it in the text, the model wrote a corrected request: $38. That passes the check.

Trip 4 · Two requests in one reply

Added to the package
That request + “Refund of $38 succeeded.”
The reply
Two action requests: email Priya, leave a team note
Label
action requested
Tokens read · written
1,610 · 142

Emailing and leaving a note don’t depend on each other, so the model asked for both in one reply. The program did both.

Trip 5 · Finished

Added to the package
Those requests + “Email sent.” + “Note saved.”
The reply
“Refunded the duplicate $38.00 charge, emailed Priya, and noted the ticket.”
Label
finished
Tokens read · written
1,823 · 29

No action requested, and the label says finished. The program stops.

All five trips at a glance

The five trips: what was added, the reply, its label, and tokens read and written (illustrative counts)
TripAdded to the package before sendingThe replyLabelTokens read · written
1Priya’s messageAction request: look up order 4471action requested1,165 · 96
2That request + the order details (two $38 charges)Action request: refund $76 on the second chargeaction requested1,341 · 71
3That request + “Refund of $76 rejected: that charge has $38 left. Nothing was refunded.”“Only the duplicate charge should be refunded, so the amount is 38.00.” + refund $38action requested1,468 · 88
4That request + “Refund of $38 succeeded.”Two requests: email Priya, leave a team noteaction requested1,610 · 142
5Those requests + “Email sent.” + “Note saved.”“Refunded the duplicate $38.00 charge, emailed Priya, and noted the ticket.”finished1,823 · 29

The tally (illustrative counts, as noted above): 5 trips, each carrying the whole history. “Tokens read” went from 1,165 to 1,823 even though Priya’s message never changed. About 7,400 tokens read and 430 written in total, for one support message.

See this with code

What the model really reads

The model doesn’t see “three parts”. Before the formula runs, the model service flattens the whole package into one long piece of text. Little markers say who is speaking: the instructions, the user side, the assistant (the model’s own turns). Then that text is chopped into tokens, exactly as on page 1.

Every company’s markers are different, and Claude’s aren’t published. So here is trip 3 flattened by a real open model (Qwen3), shown with plain-English labels:

[start: instructions]<|im_start|>system You are the support agent for Tidy Home, an online store. …
… the action menu, pasted in as text … [end]<|im_end|>
[start: user]<|im_start|>user Hi, I got charged twice for order #4471. Can you sort it out? [end]<|im_end|>
[start: assistant]<|im_start|>assistant I’ll pull up order #4471 first.
[action request]<tool_call> look up order 4471 [end request]</tool_call> [end]<|im_end|>
[start: user]<|im_start|>user [result]<tool_response> order 4471, total 38, two charges of 38… [end result]</tool_response> [end]<|im_end|>
[start: assistant]<|im_start|>assistant [action request]<tool_call> refund 76 on charge ch_8813 [end request]</tool_call> [end]<|im_end|>
[start: user]<|im_start|>user [result]<tool_response> Refund of 76.00 rejected… [end result]</tool_response> [end]<|im_end|>
[start: assistant]<|im_start|>assistant ▌

“…” marks cuts. Action details are shown in plain words here; the real text writes them in a set format. ch_8813 is the store’s reference number for the second charge.

Show the real markers
  • <|im_start|>system = [start: instructions]
  • <|im_start|>user = [start: user]
  • <|im_start|>assistant = [start: assistant]
  • <|im_end|> = [end]
  • <tool_call> … </tool_call> = [action request] … [end request]
  • <tool_response> … </tool_response> = [result] … [end result]

They look strange, but they’re tokens like any other. “<tool_call>” is token 151657.

What to notice

  • It’s one long row of numbers. In Qwen3, trip 3 comes to 780 tokens, and every next piece the model writes is computed from all 780. (The trip table above says 1,468 for the same trip. Those counts are illustrative, for a different model with its own list of pieces and its own hidden text about actions. Same idea, different count.)
  • The action menu is only text. It’s pasted into the instructions. In training, the model saw many examples of what text to write when one of the listed actions fits.
  • An action request is text the model wrote. The model service spots the action-shaped part of the reply and hands it to the program as a tidy request with a reference number.
  • Results come back on the “user” side. “User” means everyone who isn’t the model: the program’s reports arrive as another user turn, wrapped in [result] markers so they look different from the customer’s words.
  • The text ends on an open line for the assistant. Whatever the model writes next is its turn.
  • Numbers are written one digit at a time. In this model, the amount 38.00 is five pieces: “3”, “8”, “.”, “0”, “0”. The model writes each one as a likely next piece. Nothing in it works like a calculator. That’s one more reason the program checks every amount.

See this with code

Why long jobs cost more

Model services charge by the token: some amount for every token the model reads, and usually more for every token it writes.

Because the whole package is re-sent every trip, the reading adds up fast. Priya’s message is about 20 tokens. The job read about 7,400.

Tokens read on each trip of Priya’s job (illustrative counts)
TripRead this tripWrittenRead so far
11,165961,165
21,341712,506
31,468883,974
41,6101425,584
51,823297,407
Total7,4074267,407

So:

  • Longer jobs cost more per step. Each trip carries everything before it.
  • Longer jobs get slower. More to read before the first new piece comes out.
  • There’s a ceiling. The context window is the most tokens a model can take in at once. A long job, or a long chat, will eventually fill it. Past that, older parts must be cut or summarized.

What the services and programs do about it

  • Reuse the unchanged start. The instructions and menu are the same every trip. The program still sends the whole package every time, but the service can recognize a start it has seen in the last few minutes, reuse its work on it, and charge less to re-read it. The model still takes all of it into account; only the price and the wait go down. (You’ll see this called prompt caching.)
  • Summarize old parts. The program replaces early parts of the conversation with a short summary. Cheaper, but details can be lost.
  • Set a spending limit per job, so nothing runs up a bill unattended.

See this with code (and the tally under the round trip)

The safety checks are ordinary rules

The standing instructions told the model never to refund more than the charge. It asked for $76 anyway. That’s normal: instructions are requests a model usually follows, not guarantees.

The guarantee came from the program. Before carrying out any action request, it runs checks: rules in the program, written by people the old-fashioned way, that every action request must pass.

  • Is this action on the menu? Anything else is refused.
  • Are the details allowed? A refund can’t be more than what’s left on the charge.
  • Step limit. The step limit is the most times the program will ask the model during one job. After that (10 in the example program), it stops and hands the job to a person.
  • Spending and time limits. Stop if a job costs too much or runs too long.
  • Ask a person first. For risky actions, like large refunds, pause until someone approves.

The model’s request

Refund $76 on the second charge

  1. On the menu? Yes: “Refund a charge”.
  2. Details allowed? No: $76 is more than the $38 left on that charge.

Refused. Nothing was refunded.

Trip 2’s request at the program’s checks.

When a check says no, nothing breaks. The refusal goes back into the package as words, like the $76 rejection, and the next prediction takes it into account.

The model suggests. The program decides.

See this with code (and what real programs add)

The short version

An agent’s conversation with a model is a package of text sent over and over: standing instructions, an action menu, and everything so far. What comes back is more text, sometimes naming an action. The program checks it, carries it out, adds the result, and sends the whole thing again, paying to re-read it each time, until the model says it’s finished or a limit is hit.

You can now explain:

  • The three players: program, model service, model.
  • The three parts of every package, and why all of it is re-sent every time.
  • What comes back, and why nothing has happened yet when it arrives.
  • How the $76 mistake was caught, and how the refusal helped the next step.
  • What the model actually reads: one long row of tokens, with markers for who’s speaking.
  • Why long jobs cost more, get slower, and eventually hit a ceiling.
  • Where safety really comes from: checks in the program, not lines in the instructions.

Next: Notes vs programs If instructions are only requests, what should go in the instructions, and what belongs in the program?

Words used on this page

Model service
Computers run by an AI company that run the model and send back its reply.
Prompt
Everything sent to the model in one go: instructions, the conversation so far, and anything else added.
Standing instructions (system prompt)
Directions written by whoever built the app, sent at the top of every request.
Action menu
The list of allowed actions, each with a short description, sent to the model every time.
Action request
Text the model writes that names an action and its details, like “look up order 4471”; nothing happens until the program carries it out.
Context
Everything the model is shown on one request: the whole note it reads before predicting. (In everyday talk, “prompt” and “context” often mean the same thing.)
Context window
The most tokens a model can take in at once; past that, older parts must be cut or summarized.
Token
A small piece of text, such as a word, part of a word, a space plus a word, a single digit or a punctuation mark.
Check
A rule in the program, written by people, that every action request must pass before it is carried out.
Step limit
The most times the program will ask the model during one job before it stops and hands over to a person.
Memory (in AI apps)
Notes that the app saves and re-sends; the model itself keeps nothing between requests.

Sources