Your Coding Agent Is Failing Because of a 3-Line Config Change You Haven’t Made
Your Coding Agent Is Failing Because of a 3-Line Config Change You Haven’t Made
At temperature 2, the most popular sampling method resolves zero GitHub issues. Zero. The same model, same weights, different sampler: 46–62 issues solved. If you’re running agentic workflows in production and haven’t touched your sampler settings, you may be leaving half your agent’s capability on the floor.
What happened
Researchers at tested 29 open-weight models (1B to 122B parameters) across 10 benchmarks plus needle retrieval out to one million tokens, covering agentic workflows, long chain-of-thought reasoning, long-form writing, and structured output. The core finding: at elevated temperature, the sampler — the rule that picks the next token from the model’s probability distribution — is the decisive variable in whether long-generation tasks succeed or collapse. The culprit is pool size: going from temperature 1 to temperature 2, top-p’s candidate token pool explodes from 4 tokens to 51,845 on a Llama-3.1-8B story task, while top-nsigma stays at 3. Samplers that contain the pool (top-nsigma) keep a median 100% of their temperature-1 score as heat rises; top-p retains only 26%. Across 133 sampler-setting-and-temperature combinations, pool size predicts output quality with a Spearman correlation of −0.77.
Cold read
The headline numbers (0 vs. 46–62 issues on SWE-bench) are striking, but they’re produced at temperature 2 — a setting most production deployments don’t use. At the default temperature of 1, the paper itself concedes “most samplers perform about the same,” which significantly narrows the operational urgency. The benchmarks are real and varied, but the paper tests open-weight models only; closed-API models (GPT-4o, Claude, Gemini) have their samplers locked away, so you can’t replicate the fix on the systems most founders actually pay for. Top-nsigma is not a universally available option in mainstream inference stacks — it’s not a dropdown in LangChain or a parameter in the OpenAI-compatible API. And the benchmark contamination risk on SWE-bench Verified is non-trivial given how widely it’s been used for training signal in 2025–2026.
What it means for you
- Signal maturity: 3/5 — Strong empirical coverage, but narrow real-world applicability at default temperature
- Who gets hurt: Founders running self-hosted open-weight coding agents or long-document pipelines who’ve blindly inherited top-p defaults from tutorial code
- What breaks if this is true: The conventional wisdom that “model quality = benchmark quality” collapses — you could be running a 70B model at a fraction of its capability because of a sampler config nobody on your team has looked at
- Why it might not land: Most production inference happens at temperature ≤ 1 through closed APIs where you can’t change the sampler anyway; the upgrade path is gated behind self-hosting expertise
- Watch for: Top-nsigma appearing as a named option in vLLM, llama.cpp, or Ollama release notes — that’s the moment this paper becomes a one-line production win
Forecast as of 2026-11-18
By Q3 2027, at least one major open-source inference framework (vLLM or llama.cpp) will ship top-nsigma as a first-class sampler option, and at least one hosted open-weight inference provider (Fireworks, Together, or equivalent) will expose it via API — or this paper will have been quietly buried because the gains don’t replicate outside controlled conditions.
Source: Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat) — Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv, Judah Goldfeder, Sanjay Basu. https://arxiv.org/abs/2611.22148v1
