Trimming old tool results out of a long LLM conversation is one of the most-cited ways to cut token spend on agentic workloads. Measure it naively and the savings look large: on our own corpus of agent traffic, evicting stale tool results removes about 44% of replayed message content. The problem is that the number is mostly an illusion. Once you price in the provider’s prompt cache, that 44% collapses to about 13% of actual cost. The same corpus, with a slightly different eviction strategy, keeps 36%. The gap between those two outcomes is the whole story, and it is one a client-side tool cannot see.
Where the tokens actually are
We measured a corpus of 60 real agentic-coding sessions (11,049 API calls) drawn from our own development, exactly the workload this kind of compression targets. Two facts stand out. First, 46% of appended message content is tool results, and 92% of those bytes come from just two sources: shell output and file reads. That is a large, concentrated target. Second, real traffic ran a 99% prompt-cache hit rate. Almost every request re-sent a context the provider had already cached and was billing at a fraction of the full rate. Both facts matter, and the second is the one most writing on compression ignores.
The saving that survives caching
Here is the finding in one picture. Two eviction strategies, each shown as its nominal token saving next to the actual cost saving once the provider’s cache is priced in.
The mechanism is the prompt cache, and specifically the prefix. A provider caches the longest unchanged leading span of your context and bills it cheaply next time. Eviction rewrites history, and the moment you edit a message inside the cached prefix, everything after it is invalidated and re-billed at full price. Evict a little every turn and you break the prefix constantly; batch the eviction, waiting until the context crosses a size threshold before dropping a chunk, and you break it rarely.
The full comparison
Every candidate we measured, with cost shown both on a cache-supporting provider and on one without caching. The nominal column is what a token-count-only tool would advertise; the cached column is what you would actually pay.
| Strategy | Nominal input saved | Cost saved (cached provider) | Cost saved (no cache) |
|---|---|---|---|
| Exact dedup of repeated tool results | 0.2% | 0.2% | 0.2% |
| Sliding eviction, keep last 3 | 45.2% | 34.1% | 45.2% |
| Sliding eviction, keep last 8 | 44.2% | 13.1% | 44.2% |
| Batched threshold eviction | 38.8% | 36.3% | 38.8% |
| Dedup + batched combined | 38.8% | 36.3% | 38.8% |
Two things to read off it. Nominal and no-cache columns are identical, because without a cache every removed token is a token you stop paying for. And the sliding rows swing wildly on the cached column for the same nominal saving: keeping the last 8 collapses to 13.1% while keeping the last 3 holds 34.1%, because how much history you disturb per turn changes how often the cache breaks. Batched eviction sidesteps that knob entirely and lands at 36.3% regardless. Exact dedup is near zero here only because a well-engineered client already avoids re-sending identical results; a naive agent loop would show far more.
Why not just truncate harder?
A tempting shortcut is to shorten each tool result in place, keep the first and last few lines, drop the middle, rather than evicting whole stale ones. For code, the evidence says don’t. Load-bearing detail (error messages, file paths, failing test names, exact identifiers) is fine-grained and interleaved with the redundant text, so first-N/last-N truncation cuts some of it every time. In an independent terminal-agent benchmark, a fixed-rule compressor of exactly this class scored below doing nothing at all:
Why this is a gateway’s job
Picking the right strategy requires knowing three things at request time: which provider is about to serve the request, whether it supports prompt caching, and what its cache price ratios are. A client-side library sees none of them cleanly, it sits above the routing decision. A gateway sits exactly at it. It can pick batched eviction where caching applies and a more aggressive strategy where it does not, and quote savings that are cache-adjusted rather than nominal, per provider and per request. That is also the discipline we hold ourselves to: account for total tokens, adjust for the cache, and verify that output quality actually held rather than assuming it did. On code workloads especially, a saving that quietly degrades the model is not a saving.
The caveats, stated plainly
These numbers come from a single client (one coding agent) against one provider family, with cache economics modelled on that provider’s published read and write ratios and token counts approximated from character length. The percentages are of message content, which is roughly a third of total context here, so the end-to-end bill impact on this heavily-cached corpus is closer to 12–13%, rising toward the nominal figure on providers without caching. Your traffic will differ: a naive agent loop that re-sends identical results will have far more headroom, a well-engineered client far less. The point is not the specific percentage. It is that you cannot know your own number without measuring it against your real provider and its cache, which is the measurement worth doing before you trust any compression claim, including ours.