AI agents: automating a process without breaking what already works
4 min read
Key takeaways
- The costliest item is almost always the context sent, not the response generated.
- A cache on repeated instructions sharply reduces the cost of similar calls.
- Using the most powerful model for every task multiplies the bill with no measurable gain.
- Set a spending cap and an alert before going to production, not after the first incident.
- Measure the cost per useful business action, not the cost per call.
A company calls us because its AI bill has been multiplied by more than twenty in a month. No change in pricing, no particular usage peak: a code change had caused the full conversation history to be sent with every request instead of the last three exchanges.
Bills that run away are almost never a pricing problem. They are design problems, and they can be fixed in a few days.
Understanding what you pay for
You pay for the text that goes in and the text that comes out, counted in units called tokens — roughly, a common word is worth one to two tokens in French.
Two findings follow, and they are counter-intuitive.
Input costs less than output per unit, but it often makes up most of the bill, because you send a lot and receive little. A document assistant commonly sends 6,000 tokens of context to produce 250 tokens of response.
The cost per call is trivial; it is the volume that counts. A tiny cost per request seems negligible until forty people make three hundred a day.
The five mechanisms that make a bill run away
1. Context sent every time
Culprit number one. The entire history, or twenty document passages “just in case”, is sent with every request.
The fix: send only what is necessary. On a document assistant, going from twenty passages to five — after a reranking step that selects the best — sharply reduces cost without degrading quality, often improving it. This is the mechanism described in our article on RAG assistants.
2. No cache on instructions
Most applications send the same two thousand tokens of system instructions with every request. Providers offer caching mechanisms that bill this part at a reduced rate when it is reused.
The fix: place the stable part — instructions, examples, reference documents — at the start of the request and enable the cache. It is a change of a few hours for a permanent gain.
3. The oversized model
Using the most capable model to sort a request into three categories is like taking a lorry to deliver a letter.
The fix: route by task. A lightweight model to classify, extract, rephrase; the capable model to reason, synthesise, write. The price difference between tiers is counted in multiples, not percentages.
4. Cascading calls
An agent that chains eight steps makes eight calls, each carrying the accumulated context of the previous ones. Cost then grows quadratically with the number of steps.
The fix: limit the number of steps, summarise the context between steps rather than accumulating it, and set an iteration cap. It is also what prevents infinite loops — the subject of our article on AI agents.
5. No safeguards
No limit per user, no account cap, no alert. A badly written loop in production can then consume a monthly budget in a single night.
The three safeguards to put in place before going to production: a spending cap on the provider account, an alert at 50% and 80% of the monthly budget, and a limit on calls per user per hour in your application. One day of work, and no surprise is possible any more.
The order in which to optimise
1stReduce the context sent: the biggest lever, almost always
2ndEnable the cache on the stable part of requests
3rdRoute simple tasks to a lighter model
These three levers, applied together, commonly divide a bill by three to five with no measurable loss of quality. We apply them systematically before considering anything else.
Finer optimisations — quantisation, self-hosted models, batch processing — only make sense at high volumes, and they introduce an operational burden you must be able to carry.
The right indicator: cost per useful action
Tracking total monthly cost tells you nothing. Tracking the cost per successful business action tells you everything.
Examples: cost per case processed, cost per response validated without rework, cost per document correctly extracted. This figure can be compared with what the action used to cost in human time — and it is the only argument that holds up in a budget committee.
On a healthy use case, you commonly get a cost per action of a few cents against several euros of human time saved. The debate over API pricing then stops by itself.
An AI bill that causes worry is almost always the symptom of a system sending too much, too often, to the most expensive model. It is not a supplier problem, it is an engineering problem — and therefore fixable.
The check in three questions
To put to your team or your provider, once a quarter:
- How many tokens are sent per request on average, and why? If nobody knows, that is the first job.
- What proportion of requests benefits from the cache? Below 50% on an application with stable instructions, there is an immediate gain.
- Which tasks use the most expensive model, and did they really earn it? A one-day comparative test is enough to settle it.
Sources
- Understanding and counting tokens — OpenAI
- Pricing — OpenAI
- Prompt caching — Anthropic
- Prompt caching — OpenAI
