There is a persistent assumption inside enterprise AI teams that more context is safer. If you are not sure the model has what it needs, stuff more into the prompt: the full document, the whole conversation history, every retrieved chunk, the entire system instruction set. Tokens are cheap and getting cheaper, so why economize? The problem is that this instinct is wrong on two fronts at once. Bigger prompts cost more, and they measurably make the model worse at its job. For anyone accountable for AI spend or AI reliability, that is a combination worth taking seriously.

Bigger prompts answer worse

The research here is unusually clear. Models do not read long contexts evenly. Accuracy is highest when the relevant information sits at the very beginning or the very end of the prompt and drops sharply when it lands in the middle, a U-shaped curve that holds even for models explicitly built for long context (Liu et al., “Lost in the Middle,” TACL 2024 ). Bury the one fact that matters in the middle of a 40-page dump, and you have quietly lowered your odds of a correct answer.

It gets worse as the window fills. On a benchmark designed to test comprehension rather than keyword matching, 11 models fell below half of their short-context accuracy at 32K tokens. Even GPT-4o, one of the strongest performers, dropped from an almost-perfect 99.3% to 69.7% (Modarressi et al., “NoLiMa,” ICML 2025 ). And the degradation is not only about length. It is about noise. Adding plainly irrelevant text to a reasoning prompt dramatically reduces accuracy on its own, independent of how many tokens the model can technically hold (Shi et al., Google, ICML 2023 ).

The practical takeaway for a platform owner: context bloat is not a neutral safety margin. It is an active reliability risk. The extra material you added “just in case” is distracting the model from the material that mattered.

Bigger prompts cost more, and the bill is not where you think

The counterargument is that inference keeps getting cheaper, and it does. The cost of running a GPT-3.5-level system dropped over 280-fold between November 2022 and October 2024 (Stanford HAI, 2025 AI Index ). But unit price falling is exactly what lulls teams into wasting units. When each token is nearly free, nobody counts them, until you multiply by every request across every agent, every day.

That is why the cost story does not end at the price sheet. Gartner projects that through 2028, at least half of GenAI projects will overrun their budgets, not because model pricing spiked, but because of poor architectural choices and a lack of operational know-how (reported via PureAI ). Prompt bloat is precisely that kind of choice: an easy default that compounds invisibly at scale.

There is an adversarial dimension too. OWASP’s 2025 LLM Top 10 elevated “Unbounded Consumption,” including “denial of wallet” attacks that exploit pay-per-token billing by driving high volumes of expensive operations against you (OWASP LLM10:2025 ). If your prompts are already oversized by default, an attacker inflating volume is amplifying a cost surface you built yourself.

Most of that context is dead weight

Here is the part that reframes the problem. If long prompts were dense with essential information, the tradeoff would at least be real. They usually are not. Prompt-compression research has shown that heavily padded prompts can be shrunk up to 20x with minimal quality loss (peer-reviewed study ). In some cases, trimming the fat does more than hold quality steady. One study improved answer accuracy by up to 21.4% while using roughly 4x fewer tokens (ACL 2024 research on long-context prompts ).

Read those two findings together and the conclusion is uncomfortable: the typical enterprise prompt is mostly non-essential tokens. You are paying to send them, paying to have them degrade the answer, and paying again on every retry the bad answer triggers.

What good looks like

The fix is not “write shorter prompts” as a coding guideline, that does not survive contact with dozens of teams, RAG pipelines, and agent frameworks each assembling context their own way. The fix is treating context as something to be measured and governed at the boundary, not trusted by default. In practice that means:

  • Visibility. Someone should be able to answer “how big are our prompts, by team and by route, and are they growing?” That question is usually unanswerable today.
  • Enforcement at the edge. Trimming and cost controls belong at the AI traffic layer, applied uniformly, rather than reimplemented inconsistently in every application.
  • A cost surface you can defend. Denial-of-wallet and runaway spend are the same problem viewed from two angles; both need a control point that sees aggregate traffic.

This is the role Milgram plays: a transparent proxy at the boundary between your AI applications and their model providers. Because it sits in the path of every request, it can see and act on the traffic that individual apps cannot: compressing bloated prompts before they reach the provider, and giving platform owners a single place to measure and control what leaves the building. No application rewrites, no per-team discipline to enforce by hand.

Context is not free, and it is not neutral. The teams that treat the AI traffic boundary as a control layer (rather than a passthrough) are the ones who will keep both their bills and their answers under control. You can see how we approach it at milgram.dev .