Across roughly 970,000 inferences to date, Milgram has removed about 481 million tokens before they ever reached the model provider: 54.6% of the original token volume. That is the number that matters when the conversation turns to prompt compression, LLM inference costs, and token reduction, because AI cost optimization is not a pricing negotiation. It is an engineering problem, and the tokens you never send are the cheapest ones you will ever pay for.
The sticker price of a token keeps falling. Enterprise LLM spend does not. Menlo Ventures’ 2025 enterprise report put annualized enterprise spend on large language models at $8.4 billion, more than double the prior year . Cheaper units, larger bills. The gap is volume, and volume is the one variable most teams have left completely untouched.
Why LLM inference costs scale the way they do
Every major provider bills the same way: per token, metered on the way in and on the way out. Two properties of that model decide your bill.
First, output costs more than input. On Anthropic’s published schedule, Claude Opus 4.8 is $5 per million input tokens versus $25 per million output tokens , a 5x asymmetry that holds across the model line. Claude API costs and OpenAI API costs both punish verbosity on the response side hardest.
Second, repeated context compounds. Agent loops re-send the same system prompt, tool schemas, and history on every turn. Prompt caching helps: a cached-content read is billed at 10% of the standard input price , a 90% discount on the repeated part. But caching only rewards bytes you keep sending. It does nothing about the bytes that never needed to be there.
That is the token usage most teams cannot see. The provider invoice tells you what you spent, not what you wasted.
What prompt compression research actually shows
Prompt compression is a real, measured field, not a marketing claim. The peer-reviewed literature splits it into hard methods, which drop low-information natural-language tokens while keeping readable text, and soft methods, which learn compact embedding-space representations . The published benchmarks are striking:
- LLMLingua reports up to 20x compression with little performance loss ; Microsoft Research’s write-up shows a 2,366-token prompt compressed to 117 tokens with only a 1.5-point drop on GSM8K .
- LongLLMLingua goes further and shows compression is not always a tax on quality: it boosts performance by up to 21.4% while using around 4x fewer tokens, and cuts cost by 94.0% on the LooGLE benchmark .
- Extractive compression on long contexts reaches up to 10x with minimal accuracy degradation , often beating both summarization and naive token pruning.
- At the far end, learned approaches like 500xCompressor report 6x to 480x ratios while retaining 62% to 73% of uncompressed capability .
This is the honest answer to the question enterprises keep asking, “how to reduce LLM inference costs without degrading output quality”: the research says it is achievable, within limits, and the limits are measurable. One caveat worth stating plainly. An independent study, Prompt Compression in the Wild , found that while response quality stayed statistically unchanged for summarization, code generation, and question answering, end-to-end latency gains are conditional and can be erased by optimized serving stacks. The robust, portable win is token and cost reduction. Speed is situational. We say so because pretending otherwise is how vendors lose trust.
Compressed vs uncompressed: the token math
Compression reframes prompt compression vs context window optimization. A bigger context window lets you send more; compression asks whether you should. Here is the aggregate, using Milgram’s own production traffic and an illustrative input rate.
| Metric | Uncompressed | With Milgram | Reduction |
|---|---|---|---|
| Aggregate tokens (to date) | 881.9M | 400.5M | 54.6% |
| Per 1M tokens of traffic | 1,000,000 | ~454,000 | 54.6% |
| Illustrative cost @ $3 / 1M input tokens | $3.00 | $1.36 | $1.64 saved |
The percentage is the point. Whatever your monthly token usage, roughly half of it, on this workload profile, was structural overhead that added nothing to the model’s answer. Compression is the difference between paying for signal and paying for scaffolding.
Where compression belongs: the AI traffic boundary
Most cost tooling sits beside the problem: dashboards that tell you what you already spent. Compression has to sit in the path. Milgram is a transparent proxy at the AI traffic boundary. Every request to every provider flows through it, and token reduction happens there, before the request leaves your perimeter, with no application rewrite and no change to the model you call.
That placement is what makes it an LLM firewall rather than a report. The same choke point that inspects traffic for prompt filtering and data-leak risk is the natural place to shrink it. An AI gateway that only routes and logs leaves the tokens untouched. A control plane that sits inline can act on them. The specific mechanics of how Milgram compresses are ours to keep, but the outcome is not a secret: 54.6% removed, in production, without a quality regime change on the caller’s side.
Token reduction is also an attack-surface control
Cost and security are the same axis here, and the standards bodies now say so directly. The OWASP Top 10 for LLM Applications added LLM10:2025 Unbounded Consumption, which names uncontrolled inference volume as a Top-10 risk driving economic loss and service degradation . It even coins a term for the failure mode: Denial of Wallet, where attackers exploit pay-per-token pricing to impose unsustainable financial burden . Sitting one rung above the cost risk is LLM02:2025 Sensitive Information Disclosure , the second-ranked risk on the list, and every token you stop from leaving is a token that cannot leak.
A boundary that measures and reduces token volume is therefore doing two jobs at once: it is enterprise LLM cost management and it is a consumption control against a named OWASP risk. The FinOps Foundation’s data confirms the urgency. The share of organizations actively managing AI spend climbed from 31% in 2024 to 63% in 2025 to 98% in 2026, with “FinOps for AI” now the top forward-looking priority . Nearly every enterprise is now watching this line item. Very few are instrumented to act on it at the point where the tokens are actually created.
The takeaway
Prompt compression is not a trick for shaving a rounding error. It is a structural lever on LLM inference costs, backed by peer-reviewed benchmarks and, in Milgram’s case, by more than 481 million tokens removed from real production traffic. The cheapest token is the one you never send, and the only place to reliably not send it is inline, at the AI traffic boundary, before the meter starts. That is where cost optimization, data-leak prevention, and consumption control stop being three tools and become one.