You Can Now Train a Math Model Without the Answer Key—Almost

You Can Now Train a Math Model Without the Answer Key—Almost

The answer key is the holy grail of RL training. Without it, your reward signal is noise and your model learns to game the judge. Two researchers just showed a calibrated LLM judge can match ground-truth supervision on a math benchmark—which sounds like a jailbreak for annotation costs, until you read the fine print.

What happened

Agnihotri and Kapoor introduce RLCDR: instead of using exact-match checkers (which need reference answers) or standard LLM-as-judge scores (which are uncalibrated confidence numbers dressed up as probabilities), they convert an LLM into a decision model that outputs a proper probability distribution over typed questions like “is this answer correct?” The key operations: single forward pass readout, debiasing for option-order effects, and calibration on roughly 100 labels. They train Qwen3-1.7B with GRPO on MATH and hit 76.8% on MATH500—up from a 72.2% baseline, a +4.65 point gain (p < 1e-4)—essentially matching the 76.9% you’d get training on the answer key directly. A random reward only moved the needle +0.65 points, which is the right control to run. The harder the problem, the bigger the gap: on the hardest difficulty tier the model jumps from 44.8% to 51.7%. On calibration specifically, the method’s expected calibration error is 0.074 versus 0.141 for a standard softmax judge on the same 27B backbone—half the error. As a bonus, the same reward model outperforms a dedicated 7B process reward model on step-level error detection (ProcessBench F1: 0.772 vs. 0.722).

Cold read

The match with answer-key performance is real but narrow: it holds on the MATH distribution where the base model already has reasonable coverage (72.2% baseline). The authors themselves flag the breakdown case—when trained on problems the base model mostly fails, the reference-free reward only gets you +1.9 points and trails the answer key, because the judge becomes overconfident on out-of-distribution difficulty. That’s the exact regime where you’d most want to avoid buying labeled data. The benchmark contamination concern is also non-trivial: Qwen3 models have checkered data provenance histories and MATH500 is a well-worn eval. The calibration was achieved with ~100 labels, which sounds cheap, but those labels need to be representative—if your domain shifts (new problem types, new response styles), recalibration costs are recurring, not one-time. Finally, this is one model family, one math benchmark, one RL algorithm (GRPO); generalization to code, reasoning over documents, or instruction-following is completely untested.

What it means for you

  • Signal maturity: 2/5 — Single-domain proof of concept, one architecture, known-hard benchmark with contamination risk
  • Who gets hurt: Annotation vendors and golden dataset pipeline builders selling “RL-ready labeled data” for math and STEM tutoring products
  • What breaks if this is true: The assumption that reference-answer supervision is load-bearing for RL fine-tuning collapses in mid-difficulty domains—annotation budgets get reallocated to calibration data, not answer keys
  • Why it might not land: The method degrades exactly where you need it most (hard, low-coverage problems), and calibration maintenance on a moving policy is an engineering overhead most small teams will underestimate
  • Watch for: A replication on code generation (HumanEval/LiveCodeBench) with similar +4–5 point gains without test-case oracles—that would be the signal this generalizes beyond math

Forecast as of 2026-11-21

By Q3 2027, at least two open-source RL training frameworks (e.g., TRL, OpenRLHF) will have merged a calibrated-judge reward option, but adoption for production fine-tuning outside math/STEM domains will remain below 15% of teams using RL—the out-of-distribution overconfidence problem won’t be solved fast enough to change default practice.


Source: Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is — Rudransh Agnihotri, Manasvi Kapoor. https://arxiv.org/abs/2611.25135v1

Similar Posts