Glonce

Fine-tuned 4B Qwen 3.5 beats GPT-5.6 on MetroLLM-Bench

researchThe Glonce editorsHugging Face🔥67Glonce rating67Impact55Novelty65Relevance90Surprise60Players40Credibility85
Fine-tuned 4B Qwen 3.5 beats GPT-5.6 on MetroLLM-Bench

MetroLLM-Bench is a new 955-case benchmark that tests language models as the policy layer of a transit ticket kiosk, the component that decides how the kiosk should respond. It covers six real metro systems ranging from 37 to 414 stations and eleven task categories, including routing, fare calculation, disruptions, accessibility, and adversarial input. In every case the model must call structured tools and return a machine-renderable terminal state: an outcome, a per-ticket fare quote where one applies, and a kiosk action.

Scoring runs on two tiers. Fourteen deterministic components make up Tier 1; eight semantic-quality components make up Tier 2, six of which are themselves judged by another language model. The paper reports the Tier 1 score alone and the combined score across both tiers. The 955 cases are split 75/25: 717 go toward generating training data, and 238 are held out for evaluation.

The authors evaluate 26 models from six vendors, of which 23 are ranked. The headline result: on the 238-case held-out set, a 4B-parameter Qwen 3.5 model fine-tuned with parameter-efficient fine-tuning (PEFT) scores 91.3 on Tier 1, ahead of both tested GPT-5.6 configurations, which score 90.6 and 90.0 (the source does not say which configuration is which), and matches GPT-5.4 full running at maximum reasoning effort (91.4). The fine-tuned model has a 2.6 GB footprint at Q4_K_M quantization.

Bigger does not help here, at least not at this training scale: larger 9B and 27B PEFT students score no higher on Tier 1 than the 4B one. Measured against each size's own un-tuned base model, the PEFT gain falls from +7.03 points at 2B parameters, across three training seeds, to a slight loss of -0.91 points at 27B; every seed points the same direction at every size. A plain deterministic rule-based baseline, with no learned model at all, already reaches 84.6 on Tier 1; the authors say the remaining advantage language models hold over it is concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning.

The Qwen student's Tier 1 lead does not carry over to the paper's composite ranking, which Muse Glimmer 30B leads instead. The authors also report that serving configuration alone can shift a Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points. The benchmark, its evaluation harness, a reproduction guide, and the fine-tuned Qwen students are released at github.com/continker/metrollm-bench; the source states no license for any of it, and it names neither the paper's authors and their institutions nor the six vendors behind the 26 evaluated models.

Key facts

  • MetroLLM-Bench is a new 955-case benchmark that tests language models as the decision-making layer of a transit ticket kiosk, covering six real metro systems (37 to 414 stations) and eleven task categories, including routing, fare calculation, disruptions, accessibility, and adversarial input.
  • Cases are scored on 14 deterministic Tier 1 components and 8 semantic-quality Tier 2 components, six of the latter judged by another language model; results are reported for Tier 1 alone and for the combined score, over a 75/25 split of 717 training cases and 238 held-out evaluation cases.
  • On the 238-case held-out set, a 4B-parameter Qwen 3.5 model fine-tuned with parameter-efficient fine-tuning (PEFT) scores 91.3 on Tier 1, ahead of both tested GPT-5.6 configurations (90.6 and 90.0) and matching GPT-5.4 full at maximum reasoning effort (91.4), with only a 2.6 GB footprint at Q4_K_M quantization.
  • Scaling the PEFT student further does not help at this training scale: the fine-tuning gain over each size's base model falls from +7.03 points at 2B parameters to -0.91 at 27B, and 9B/27B students score no higher on Tier 1 than the 4B one; a plain rule-based baseline already reaches 84.6 on Tier 1.
  • The Qwen student's Tier 1 lead does not carry over to the paper's composite ranking, led instead by Muse Glimmer 30B; the authors also report that serving configuration alone can shift a Qwen 3.5-to-3.8 comparison by 2.7 Tier 1 points.

Why it matters

The headline finding cuts against the assumption that a bigger, more expensive model is the safer bet for a structured task: a 4B-parameter Qwen 3.5 model, fine-tuned with parameter-efficient fine-tuning and small enough to fit a 2.6 GB footprint at Q4_K_M quantization, matches or beats GPT-5.6 and GPT-5.4 configurations on the deterministic Tier 1 score for transit-kiosk decision-making. That is a case for running a small, cheaply fine-tuned model locally instead of calling a large hosted model, at least for tasks with the kiosk's structure: a fixed set of tools, a machine-checkable output, and a bounded domain. The benchmark is also a tool-calling-centric way to evaluate models on an operational task rather than open-ended chat, and its finding that the PEFT fine-tuning gain shrinks as the base model grows, from +7.03 points at 2B parameters to a small loss at 27B, is a data point for anyone deciding how large a base model to start from before fine-tuning.

Who it affects

Teams building or buying self-service ticketing systems for transit agencies get a concrete way to compare a small, locally hosted model against a large hosted API model on the kiosk's actual decision-making: routing, fares, disruptions, accessibility requests, and malformed input. More broadly, it affects anyone evaluating language models for structured, tool-calling tasks with a checkable output, since MetroLLM-Bench is one more benchmark showing a small fine-tuned model can match a much larger general-purpose one on that kind of narrow, deterministic job. Researchers comparing parameter-efficient fine-tuning against full fine-tuning across model sizes are a third audience, given the paper's specific finding that the PEFT gain shrinks as the base model grows.

How to use it

The benchmark, its scoring harness, a reproduction guide, and the fine-tuned Qwen PEFT students are published at github.com/continker/metrollm-bench. That lets a team run the 955-case suite against its own model, reproduce the paper's numbers, or start from the released checkpoints instead of fine-tuning a small model from scratch. The source states no license for any of this, so terms of reuse are not given.

How solid is it

The evaluation spans 26 models from six vendors, though neither the vendors nor the individual models are named beyond the specific comparisons given; 23 of the 26 are ranked, and the source does not say why the other three are excluded. The central claim, that the 4B Qwen 3.5 PEFT student beats both GPT-5.6 tiers and matches GPT-5.4 full, is explicitly measured on the 238-case held-out partition; the source does not restate 'held-out' for the rule-based baseline (84.6), the 9B/27B comparison, the per-size PEFT-gain figures, or the 2.7-point serving-configuration effect, so it is not stated whether those numbers come from the same held-out slice. The two GPT-5.6 scores, 90.6 and 90.0, are given without saying which of the 'two GPT-5.6 tiers' each belongs to. The PEFT-gain trend across sizes is anchored at only two points, +7.03 at 2B and -0.91 at 27B, with the 4B and 9B gains not given as numbers, and it is not stated whether the three training seeds noted for the 2B figure also cover the other sizes.

Risks and caveats

No individual authors, institutions, or affiliations are named for the paper, and the six vendors behind the 26 evaluated models are not identified, so provenance cannot be checked from the source; no publication or release date is given either. The result that matters most for deployment, the 4B Qwen 3.5 student's Tier 1 score, measures only deterministic, machine-checked correctness: the paper's composite ranking, which weighs in the semantic-quality Tier 2 score as well, is led by Muse Glimmer 30B rather than by the Qwen student. Serving configuration alone can shift a same-family comparison, Qwen 3.5 versus 3.8, by 2.7 Tier 1 points, so rankings on this benchmark are sensitive to how a model is served, not only to its weights. No license is stated for the released benchmark, harness, or fine-tuned students.