“Talk Caveman, Save Money” Is Half Right — and the Wrong Half Will Bankrupt You

ByJohn

“Talk Caveman, Save Money” Is Half Right — and the Wrong Half Will Bankrupt You

Everyone in your Slack has forwarded the tip: drop grammar, truncate prompts, watch your API bill collapse. It’s wrong. A new controlled study finds that compressing inputs raises your costs and tanks accuracy simultaneously — the rare double-punishment. You’ve been optimizing the wrong channel.

What happened

Figure 2: Answer accuracy across the five reduction levels for all models and benchmarks. Solid bars denote input compression; hatched bars denote output compression. Significance markers indicate Wil
Figure 2: Answer accuracy across the five reduction levels for all models and benchmarks. Solid bars denote input compression; hatched bars denote output compression. Significance markers indicate Wil

Researchers at Adobe Research built CAVEWOMAN, a two-channel evaluation protocol testing what actually happens when you compress prompts versus model outputs — measuring task accuracy, realized per-item cost, and factual consistency against unconstrained baselines. They ran eight models across five datasets at five compression levels. The verdict splits cleanly by channel. Output compression works: it cuts realized cost 1.4–2.4x per model on most API models, up to 3x in the best case, and delivers across all four tested open-weight models under public-tier pricing. Input compression does the opposite: it raises net cost roughly 1.15x on average across five benchmarks, and up to 2.7x under stronger compression — because models compensate for garbled tokenization with longer responses even as accuracy collapses. The surface-text divergence finding is the ugliest number in the paper: on non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model’s own unconstrained baseline — a faithfulness failure that survives length-controlled re-scoring and multiple-comparisons correction.

Cold read

Eight models and five datasets is a reasonable sample, but we don’t know which eight models or which five datasets — the abstract doesn’t say, and that matters enormously if your production stack runs on one outlier. The cost arithmetic is framed under “public-tier pricing,” which is not what high-volume enterprise contracts look like; your 1.4–2.4x output-compression win may be smaller under negotiated rates. The “caveman style” prompt compression tested here is also a specific, informal variant of prompt engineering — it is not the same as structured compression schemes (e.g., LLMLingua-style token pruning) that your infra team might actually deploy. Most importantly: the surface-text divergence result is genuinely alarming but also genuinely ambiguous — “correct yet divergent from baseline” means the evaluation metric may be the thing that’s broken, not the model. The authors acknowledge this with length-controlled re-scoring, but the ambiguity isn’t fully resolved in what’s reported.

What it means for you

  • Signal maturity: 3/5 — solid controlled protocol, but model/dataset opacity limits direct applicability
  • Who gets hurt: Founders and prompt engineers running cost-reduction sprints by stripping down system prompts into telegraphic fragments — they’re spending more, not less
  • What breaks if this is true: Any agentic workflow that compresses upstream context to save tokens is likely inflating downstream response length, eating the savings and then some — plus producing outputs that look correct but diverge from what the model would say unconstrained, which poisons your golden dataset and any fine-tuning downstream
  • Why it might not land: Enterprise contracts with output-length caps, or architectures that hard-truncate responses, break the compensatory-verbosity mechanism entirely — in those environments, the input-compression penalty may not materialize
  • Watch for: API providers surfacing per-output-token cost controls (e.g., max_tokens as a first-class cost lever) as the operationally correct alternative to input compression — that’s the market signal that this finding has been absorbed

Forecast as of 2026-06-24

By Q4 2026, at least two major LLM API providers (OpenAI, Anthropic, or Google) will add explicit output-compression guidance or tooling to their cost-optimization documentation, while informal “caveman prompting” advice will visibly retreat from developer communities — a shift measurable by GitHub README and forum references. If input-compression tips remain dominant in popular prompt guides by December 2026, this paper failed to land.

Source: CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression — Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt. https://arxiv.org/abs/2606.24083v1

Similar Posts