LLMs Can Match Your Embedding Pipeline — and Cost You 1,400x More to Do It

ByJohn

LLMs Can Match Your Embedding Pipeline — and Cost You 1,400x More to Do It

The AI press will tell you that LLMs have made embedding models obsolete. They haven’t. What they’ve done is create a very expensive way to achieve a 0.4-point improvement on a benchmark. Before you rip out your vector pipeline, read the bill.

What happened

Figure 5: How many documents each architecture reads jointly with the query. Top: the pipeline. Bottom: its attention mask, drawn over the same N N documents at the same scale in every panel so the fo
Figure 5: How many documents each architecture reads jointly with the query. Top: the pipeline. Bottom: its attention mask, drawn over the same N N documents at the same scale in every panel so the fo

A controlled, cost-aware benchmark study put 10 LLMs (across six model families) head-to-head against 26 dedicated embedding models, ranging from 118M to 14B parameters, across 37 tasks covering classification, semantic similarity, clustering, pair classification, and retrieval. The headline finding: it’s essentially a tie. The best LLM — Gemini 3.1 Pro — scored 77.6 in aggregate; the best embedding model scored 77.2. A 0.4-point gap after all that hype. But the cost gap is grotesque: an LLM costs up to 1,431x more per benchmark pass ($154 vs. $0.11), and open LLMs ran 2.5× to 736× slower on identical hardware. The task-level breakdown matters more than the headline: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two are statistically indistinguishable on clustering, STS, and pair classification. One further finding with real budget implications: reasoning tokens account for 28–81% of LLM inference cost, yet reducing the reasoning budget actually preserved or improved retrieval quality for most models tested. Only Gemini 3.1 Pro makes it onto the Pareto frontier alongside leading embedding models — everything else is dominated on cost, quality, or both.

Cold read

This is a benchmark study, and benchmark contamination is a standing concern whenever frontier commercial models like Gemini are evaluated on widely-used academic task suites — the authors don’t claim to have ruled it out. The 37 tasks skew toward English, clean text, and relatively short inputs; the cost advantage of embedding models almost certainly widens further on long-document or multilingual workloads that better reflect real production systems. “Aggregate score” also obscures the fact that the right answer is task-dependent: a founder running a classification-heavy product and a founder building a RAG pipeline over dense technical documents are in different situations, but the aggregate number treats them the same. The $0.11 vs. $154 cost comparison is a single benchmark pass — real infrastructure economics compound at millions of queries per day, making the gap far more punishing than these numbers suggest. Finally, the paper does not evaluate latency under production load, which for user-facing applications is often the binding constraint before cost even enters the picture.

What it means for you

  • Signal maturity: 4/5 — Large-scale, multi-task, cost-inclusive; the methodology is unusually honest for this space
  • Who gets hurt: AI infrastructure startups and founders who already migrated their pipelines to LLM-based embedding on the assumption of clear quality wins — they paid the 1,431x premium for 0.4 points
  • What breaks if this is true: The “just use GPT-4 embeddings for everything” consulting playbook collapses; differentiation on embedding quality alone is not a defensible moat
  • Why it might not land: If LLM inference costs drop another order of magnitude in 12 months, the cost ratio shrinks enough that the quality-simplicity tradeoff shifts — this finding has an expiration date tied to GPU economics
  • Watch for: Any flagship embedding model (e.g., from Cohere, Voyage, or OpenAI) closing the 0.4-point gap on reasoning-heavy retrieval specifically — that’s the one task type where the LLM case currently holds

Forecast as of 2026-08-14

By Q3 2027, at least two dedicated embedding model families will publish benchmark scores matching or exceeding Gemini 3.1 Pro’s 77.6 aggregate on this study’s task suite at under $1.00 per benchmark pass, effectively eliminating the last task-specific argument for defaulting to LLMs in embedding pipelines.

Source: The Embedder’s Dilemma: LLMs Are Better, but at What Cost? — Adnan El Assadi, Niklas Muennighoff, Jinhyuk Lee. https://arxiv.org/abs/2608.12875v1

Similar Posts