Agent loops are expensive for a structural reason: every turn resends the entire conversation, and in a tool-using agent the bulk of that conversation is old tool output nobody will read again. Compressing that history is easy. Compressing it without destroying the provider's prompt cache is the part that decides whether you actually save money.
Why naive compression can cost more than it saves
Providers cache a prefix of your prompt and bill cache hits at a large discount. The cache is keyed on exact bytes from the start of the conversation: if byte 400 changes, everything after it is a miss.
That is precisely what per-turn compression does. Rewriting history each turn to keep it small changes the prefix every turn, so every request is a cache miss. You send fewer tokens and pay more for them — the compression wins on token count and loses on the bill.
The rule this design follows. History is only ever rewritten in a way that a later request reproduces byte-identically. Compression that cannot promise that is not worth doing.
How Routeplane compresses
Batched eviction, not per-turn
Nothing is rewritten until total message content crosses a trigger threshold. When it does, the oldest unprotected tool results are evicted in one batch until the total is back under target — and then history is left alone again until the next crossing.
Between crossings the prefix is byte-stable, so the provider's cache keeps hitting. In first-party measurement on agentic-coding workloads, batched eviction retained 36.3% of cost savings where per-turn eviction retained only 13% — same compression idea, nearly three times the retained benefit, purely from not breaking the cache.
Evictions are visible, not silent
An evicted tool result is replaced by a deterministic claim-ticket placeholder rather than deleted. The model can see that an observation existed and was dropped, instead of silently receiving a conversation that appears never to have made the call.
Deduplication happens on the way in
When a tool result is byte-identical to one already in history, it is replaced by a reference marker at the moment it enters — never by going back and rewriting the earlier copy. A retroactive rewrite would break the prefix; an insertion-time one cannot.
What it deliberately does not do
Whitespace and JSON normalization are the obvious next lever, and they are deferred on purpose. For code and agent workloads the exact bytes are frequently load-bearing — diffs, error text, and structural whitespace all carry meaning, and normalizing them changes what the model sees in ways that are hard to detect and easy to regret. It ships when it can be shown harmless per content type, not before.
The filters we removed
An earlier version shipped eleven heuristic truncation filters that shortened tool output automatically. They were removed, because when they were finally measured properly they performed below no compression at all on agentic-coding workloads: they discarded context the agent then had to re-derive, spending more tokens recovering than the truncation ever saved.
What remains is the part that survived measurement. We would rather ship one mechanism with evidence behind it than eleven that sound thorough.
A separate lever: latency-tolerant discounts
Compression reduces what you send. This reduces what you are charged for what you send, and it is independent — you can use either or both.
Providers sell a substantial discount (commonly around half price) for work that tolerates slower completion — OpenAI's Flex tier and the asynchronous Batch APIs. Routeplane can route eligible traffic into that lane when you opt in per request:
x-routeplane-batch: onThe important property is that this is content-identical: same model, same parameters, same output distribution. Only the latency contract changes. It is the rare cost lever that costs you nothing in answer quality, which is exactly why it is opt-in per request rather than applied silently — the trade you are accepting is time, and only you know which requests can afford it.
Availability
Token compression and the latency-tolerant lane are enterprise capabilities, enabled per tenant rather than on by default. Both are off unless your key is entitled to them, so neither can change what a request sends without that being a deliberate configuration decision.