LLM skills vary sharply by language, study finds

A new paper, titled "Skill Issue: Are Skills Language-Invariant in LLMs?", asks whether large language models keep the same practical skills when they operate in different languages, as distinct from whether they simply know different facts in each language. To isolate that question, the researchers built a multilingual extension to TextArena, a framework for text-based games, and ran a multilingual self-play setup: two instances of the same model play a game against each other, but each instance interacts through a different language interface. Because the model itself, the opponent, the rules, the game state and the available actions are all held fixed, the only thing that differs between the two sides is the language each one reasons and acts in. The team evaluated three open-weight models across eight languages and six games, chosen to cover spatial reasoning, imperfect information, resource allocation, and repeated interaction.

The result: the same model can play at markedly different strength depending on the language it is using, with systematic differences in win-loss margins, in how often it submits invalid actions, and in its overall strategic tendencies. Closer analysis ties these gaps to specific, language-linked failure types, including weaker spatial reasoning, worse card-conditioned decisions (choices that depend on cards or information the model is holding), and picking a suboptimal move even when a better one was available.

In some of the settings tested, the researchers found that changing only the language the model uses for its intermediate reasoning, while leaving the interface language unchanged, recovered much of the performance the model had lost. That points to language affecting distinct stages of a model's decision process rather than acting as one uniform penalty. The paper's authors frame the overall pattern as a measurable, major obstacle to building genuinely multilingual models, and argue that understanding these skill discrepancies is a step toward models that perform more equitably across languages.

Key facts

The study builds a multilingual extension to TextArena, a text-based game framework, to run multilingual self-play evaluations.
Two instances of the same model compete against each other in a game, each through a different language interface, while the model, opponent, rules, game state and available actions stay fixed, isolating the effect of language alone.
Three open-weight models were evaluated across eight languages and six games covering spatial reasoning, imperfect information, resource allocation, and repeated interaction.
The same model showed markedly different playing strength depending on language, with systematic differences in win-loss margins, invalid actions and strategic tendencies, plus language-specific failures in spatial reasoning, card-conditioned decisions and optimal move selection.
In some settings, changing only the model's intermediate reasoning language, not the interface language, recovered much of the lost performance, suggesting language affects distinct stages of a model's decision-making.

Why it matters

The paper's point is to separate two things that get lumped together when people say a model 'works in a language': whether it knows the same facts in that language, and whether it can execute the same skill at the same level in that language. By pitting two instances of one model against each other in a game with a fixed opponent, fixed rules and fixed available actions, the authors remove differences in knowledge and benchmark difficulty from the picture, leaving language as the one variable that changes. That design lets them show a skill gap exists even where there is no knowledge gap to blame it on. The authors call this a measurable, major roadblock in building truly multilingual models, since a model that reasons and acts inconsistently across languages cannot be called equally capable in all of them.

Who it affects

Anyone building or evaluating an AI system that has to reason, plan or take actions in more than one language, rather than just answer factual questions, has a stake in this finding. That includes teams deploying assistants or agents into non-English markets on the assumption that a model's English-language competence carries over, and researchers designing multilingual benchmarks, who may need to test skill and strategy, not only knowledge recall, to catch this kind of gap. Everyday users interacting with a model in a language other than English could be getting systematically weaker results on tasks that require reasoning or planning, for reasons that have nothing to do with the model's overall capability.

How to use it

This is a research contribution, not a product: the concrete output is a multilingual extension to TextArena that others can use to test whether a model's skill, not just its knowledge, holds up across languages, by having the model play against itself through different language interfaces. Teams evaluating a multilingual deployment could borrow that self-play approach to check a chosen model's practical reliability in the target language before shipping it. The paper also points to something worth testing when a model underperforms in a given language: in some of its settings, running the model's intermediate reasoning in a different language than the interface language recovered much of the lost performance.

How solid is it

The design is a clean way to isolate language as a variable: since the model, opponent, rules and available actions are all held identical between the two sides of the game, a difference in outcome can be attributed to the language each side used rather than to some confound in the setup. The evaluation spans three open-weight models, eight languages and six games, a reasonably broad sweep for this kind of study. That said, the abstract does not name which three models, which eight languages or which six games were used, and it gives no quantitative figures for the win-loss margins or invalid-action rates it describes, so the size of the effect cannot be judged from the material available here; that would require the full paper.

Risks and caveats

Because the abstract withholds the specific models, languages and games tested, as well as any authors, institutions or publication date, none of those specifics can be confirmed from the source text used for this retelling. The description of the gap as 'markedly different' is the authors' own qualitative characterization; no win rates, score margins or other figures are given in the source to size it. The abstract also does not say which language or languages the models performed best or worst in, only that variation exists, so no ranking of languages by reliability can be drawn from it.

“In some settings, changing only the intermediate reasoning language recovers much of the lost performance, suggesting that language can affect different stages of the decision process.”

— the paper's authors