Context Carrying Cost

Context carrying cost is the recurring price of holding content in a model's context window: tokens in the prompt × price per input token × requests. It is a carrying cost rather than a one-off because the context is re-sent and re-billed on every single request, whether or not the model needed it that turn.

Teams usually ask the wrong question first. Fifty pages of prose is roughly 33,250 tokens, comfortably inside a 128,000-token window — so it fits, and the discussion tends to stop there. But carrying those 33,250 tokens on 1,000 requests a day at three dollars per million input tokens costs about 3,000 dollars a month. Fitting is a capability question with a yes-or-no answer; carrying is an economic question whose answer is a monthly line item.

That is why context carrying cost, not window size, is the honest input to the RAG-versus-long-context decision. Retrieval adds engineering complexity and a relevance failure mode; long context adds a recurring bill that scales with traffic. Prompt caching changes the arithmetic but does not remove it, since the discount applies only to the cached share of input. The comparison is only meaningful once both options are priced at your actual request volume.