Glonce

Same numbers, different formats trip up LLMs, study finds

researchThe Glonce editorsarXiv🔥71Glonce rating71Impact60Novelty72Relevance95Surprise62Players25Credibility85

A benchmark study tests whether language models give the same answer to a word problem regardless of how the numbers in it are written. The premise: a quantity like a quarter should get the same canonical answer whether it appears as "0.25", "1/4", "25%", "a quarter", in scientific notation, or as an exactly converted unit. The researchers generated 3,600 exact-rational problems and built 8,600 prompts around them, spanning five identity-preserving transformation families, then ran the set through five open-weight systems. Scored on a single canonical phrasing after a fixed syntax audit that normalizes answer formatting without using an LLM as judge, accuracy across the five systems is high and tight: 0.969 to 0.996. But grouping each problem with all of its reworded variants into an "orbit" and checking whether the model answers every variant correctly, and answers them all the same way, tells a different story: orbit correctness falls to 0.848 to 0.981, and orbit invariance to 0.851 to 0.981. Orbits where the model is consistent but consistently wrong are rare, at most 0.003, so most of the drop is genuine inconsistency rather than models converging on a wrong answer. Part of the apparent damage turns out to be a scoring artifact rather than a reasoning failure: most of a broad strict-parser collapse traces back to multiplication-form scientific notation falling outside the grammar the evaluator was built to parse, which the authors flag as a case of evaluator interfaces masquerading as reasoning failures. One system shows a distinct problem that survives this fix: Mistral Small 4 scores 0.699 on unit-converted inputs and produces 265 errors that differ from the correct answer by an exact power of ten, a pattern the authors call a semantic pathology rather than a parsing issue. A second, separate experiment with 9,000 calls, splitting them equally between the compared setups, tests whether varying a number's representation is a useful way to build a self-consistency ensemble. It is not: on a low-error subset, this "representation consensus" fails to beat simple "paraphrase consensus" and produces substantially more false alarms. The team released an ancillary archive with the frozen benchmark, evaluation and audit records, raw consensus responses, manifests, analysis code, and a one-command paper build.

Key facts

  • The benchmark covers 3,600 exact-rational problems and 8,600 prompts across five identity-preserving transformation families (decimal, fraction, percentage, number word, scientific notation, and unit conversion), tested on five open-weight systems.
  • Accuracy on a single canonical phrasing is high across all five systems, 0.969 to 0.996, but consistency across reworded versions of the same problem (orbit correctness and orbit invariance) drops to 0.848 to 0.981.
  • Orbits where a model is consistent but consistently wrong account for at most 0.003, meaning the drop is mostly genuine inconsistency, not agreement on a wrong answer.
  • Much of a broad strict-parser failure rate is traced to the evaluator's grammar not recognizing multiplication-form scientific notation, a scoring artifact rather than a reasoning failure.
  • Mistral Small 4 scores 0.699 on unit-converted inputs and makes 265 errors off by an exact power of ten; a separate 9,000-call test finds that using different number formats for a consensus ensemble does not outperform paraphrase-based consensus and yields more false alarms.

Why it matters

Standard accuracy benchmarks test a model on one phrasing of a problem and report a single number. This study shows that number can hide a real gap: all five systems score 0.969 or above when a quantity is written in its canonical form, but their answers stop agreeing with each other once the same quantity is rewritten as a fraction, a percentage, a spelled-out word, scientific notation, or a converted unit, with orbit correctness and orbit invariance falling as low as 0.848 and 0.851. A model that is 99% accurate on decimals can still flip its answer when the input format changes, and that flip is invisible to a benchmark that only ever asks the canonical version of the question.

Who it affects

Anyone deploying LLMs on numeric word problems where inputs arrive pre-formatted in whatever way a user, spreadsheet, or upstream system happens to produce them, since the study's premise is that a percentage, a fraction and a converted unit describing the same quantity should not change the answer. It also affects people building LLM evaluation pipelines: the paper's finding that a strict-parser collapse was largely a grammar gap around multiplication-form scientific notation, not a model failure, is a direct warning about evaluator design. Mistral Small 4 users face a concrete, named issue: 265 of its errors on unit-converted inputs are off from the correct label by an exact power of ten.

How to use it

The authors published an ancillary archive alongside the paper containing the frozen benchmark, the evaluation and audit records, the raw responses from the consensus experiment, manifests, analysis code, and a one-command build of the paper itself, so the benchmark and the audit can be rerun rather than taken on faith. The paper also cautions against a specific practice: building a self-consistency check by asking a model the same question in several numeric representations. In the 9,000-call test, that approach did not beat simply paraphrasing the question and produced more false alarms.

How solid is it

The design is reasonably large for this kind of controlled study: 3,600 base problems expanded into 8,600 prompts across five transformation families, run against five open-weight systems, plus a separate 9,000-call experiment that splits calls equally between the two consensus strategies being compared. The authors also audited their own scoring pipeline rather than only their models, using a fixed syntax normalizer without an LLM judge, and explicitly separated a scoring artifact (the scientific-notation grammar gap) from the one genuine, model-specific failure they found (Mistral Small 4's power-of-ten errors). This is an arXiv preprint; the abstract gives no indication of peer-review status.

Risks and caveats

The abstract names neither the authors nor their institution, and does not identify which four systems besides Mistral Small 4 were among the five evaluated, so the exact scope of "open-weight systems" cannot be verified from the text alone. No explanation is given for why Mistral Small 4 specifically produces power-of-ten errors beyond calling it a distinct semantic pathology. The consensus result is reported for "a low-error subset" under one specific experimental setup; the abstract does not quantify how much worse representation consensus performs or say whether the finding holds outside that subset.