Glonce

TEXAS tops MoE fine-tuning baselines in 17 of 18 settings

researchThe Glonce editorsarXiv🔥68Glonce rating68Impact65Novelty70Relevance92Surprise40Players30Credibility78

Mixture-of-Experts (MoE) language models route each token to a small subset of specialized experts, and which experts fire has become a signal researchers use when adapting these models to a new downstream task. The authors argue existing approaches to finding these "task experts" have two limitations: they identify relevant experts from aggregate routing statistics that reflect how often an expert is used, not whether its use is tied to a correct answer, and they leave task-expert activations largely unexploited as a signal for deciding which parts of the training data to supervise more heavily.

The paper introduces Task-Expert-Aware Supervision (TEXAS), which addresses both gaps together. First, it discovers task experts by comparing routing activations on training instances the base model already answers correctly against instances it gets wrong, keeping the experts that activate more strongly on the successful instances rather than simply the most frequently used ones. Second, during fine-tuning, TEXAS uses that expert set to allocate supervision at the token level: it upweights the answer tokens in instances the model failed, specifically when generating those tokens activates the discovered task experts. Because the method builds on the model's existing routing behavior, it does not restrict adaptation to a fixed subset of experts or impose an explicit target distribution over routes.

The authors test TEXAS across three MoE models and six benchmarks, 18 model-benchmark settings in total. TEXAS achieves the best or tied-best result in 17 of those 18 settings and, on average, improves over the strongest baseline by 1.3 to 1.5 points. The paper reports these as relative improvements over baseline rather than absolute accuracy figures. Ablations and further analyses are presented to validate both the discovered task experts and the token-level supervision strategy built on top of them.

Key facts

  • TEXAS discovers task experts by comparing MoE routing activations on training instances the base model solves correctly versus incorrectly, keeping the experts more strongly tied to success rather than mere usage frequency.
  • During fine-tuning, TEXAS upweights answer tokens in failed instances specifically when generating those tokens activates the discovered task experts.
  • The method does not restrict adaptation to a fixed expert subset or impose an explicit target routing distribution; it builds on the model's existing routing behavior.
  • Tested across three MoE models and six benchmarks (18 settings total), TEXAS achieves best or tied-best performance in 17 of 18.
  • On average, TEXAS improves over the strongest baseline by 1.3 to 1.5 points.

Why it matters

Downstream adaptation of MoE models often leans on routing patterns to find which experts matter for a task, but prior work identifies those experts from raw usage statistics: an expert can look important simply because it fires often, regardless of whether firing it leads to a right answer. TEXAS instead conditions expert discovery on correctness, comparing activations on instances the base model solves against instances it fails, and then carries that same correctness signal into training by upweighting the specific answer tokens where the discovered experts are active. That ties expert selection and supervision allocation to the same underlying signal instead of treating them as separate, usage-driven heuristics.

Who it affects

The audience is researchers and engineers who fine-tune Mixture-of-Experts language models for specific downstream tasks, where getting more out of a fixed base model without full retraining matters. The method targets the fine-tuning stage itself rather than pretraining or inference, so it is relevant wherever teams already adapt an MoE model to a narrower task and want a more targeted training signal than uniform token-level loss.

How to use it

TEXAS is a fine-tuning-time technique: it first runs the base model over training instances to see which succeed and which fail, uses the routing activations from that pass to identify task experts, then reweights the loss on answer tokens in the failed instances according to whether the discovered experts are active there. It works on top of a model's existing router rather than requiring a fixed expert subset or a hand-set target routing distribution, which the authors present as an advantage over approaches that constrain routing directly. The source text does not name the specific MoE models or benchmarks used, nor does it mention a code or data release.

How solid is it

The evaluation spans three MoE models and six benchmarks, 18 model-benchmark settings in total, and TEXAS reaches the best or tied-best result in 17 of them, improving on the strongest baseline by 1.3 to 1.5 points on average. These are reported as relative gains over baseline; the source does not give absolute accuracy or performance numbers. The authors also report ablations and further analyses intended to validate both the expert-discovery step and the token-level supervision step separately, rather than only the combined method.

Risks and caveats

The source text is the paper's abstract-level description: it does not name the authors, their institutions, the specific MoE models or benchmarks tested, a publication or submission date, or any code or data release. The reported improvement, 1.3 to 1.5 points, is an average across settings rather than a per-benchmark figure, and no absolute performance numbers are given to gauge how strong the underlying baselines already were.