Language models retain occupational bias that tests miss

Language models often pass behavioral bias evaluations, the tests that check whether a model's outputs treat different groups of users equally. A new paper argues that passing those tests leaves an open question: has a model actually stopped associating certain users with lower competence, or has it simply learned not to say so? The authors introduce a causal framework built to answer that question directly, by separating two things that bias research usually collapses into one: a model's internal representation of a user's competence, and the model's observable output.

To probe the internal side, the authors derive steering vectors, directions inside a model's internal representations that correspond to how the model encodes a given user's expertise. They confirm these vectors are not merely correlated with behavior but causally drive it: intervening on the vectors changes what the model does, and this holds across two different settings, a question-answering task and a hiring task.

Applying the framework to several open-weight models, the authors find that demographic attributes, specifically gender, race, and socioeconomic status, shift a model's internal representation of a user's expertise. This shift shows up even in cases where the model's behavioral output reveals no measurable disparity between demographic groups, meaning a standard bias evaluation would have rated the model as unbiased. The paper also shows that intervening on these internal representations changes the model's downstream behavior, which the authors say points to failure modes that behavioral metrics alone are not able to detect.

Key facts

Language models can pass behavioral bias evaluations without it being clear whether the underlying biased associations are gone, or the model has just learned not to express them in its output.
The paper's causal framework separates occupational bias into two measurement points: a model's internal representation of a user's competence, and its observable output.
The authors derive steering vectors for a model's internal representation of user expertise and confirm the vectors causally drive behavior, tested on a question-answering task and a hiring task.
Across several open-weight models, demographic attributes, gender, race, and socioeconomic status, shift the internal competence representation even when behavioral bias tests show no disparity between groups.
Intervening on these internal representations changes downstream model behavior, which the authors say points to failure modes that behavioral bias metrics alone cannot detect.

Why it matters

The finding cuts against a common assumption in AI fairness work, that a model which passes a behavioral bias test is, for practical purposes, unbiased. This paper shows a model can score clean on exactly that kind of test while still holding a demographic-linked judgment of user competence inside its representations, a judgment that a targeted intervention can pull back out into behavior. That gap between default behavior and underlying capability is the paper's central point. It complicates any audit process that stops at checking outputs.

Who it affects

The immediate audience is researchers and engineers who build or rely on bias evaluations to certify a language model as fair before deployment; this paper argues that clearing such an evaluation is not sufficient proof on its own. It also bears on anyone deploying language models in settings where a judgment of a user's competence has real consequences, since the authors specifically test a hiring task alongside a question-answering task. The paper does not name the open-weight models it tests, so it is not possible to say from the text which specific models the findings apply to.

How to use it

This is a research contribution rather than a shipping product, and the practical takeaway is methodological. The paper's argument is that an evaluation which only checks a model's outputs can miss a bias that a representation-level probe, like the steering-vector approach used here, would catch. Teams running fairness audits can read this as a case for pairing behavioral tests with a representational check rather than treating a clean behavioral score as the end of the audit, particularly on tasks structurally similar to the paper's question-answering and hiring settings.

How solid is it

The central causal claim rests on more than correlation: the authors verify that intervening on the steering vectors actually changes model behavior, rather than only showing the vectors line up with it, and they replicate the pattern across two different task types, a question-answering task and a hiring task. The framework was also applied to several open-weight models rather than a single one, which suggests the pattern is not an artifact of one system. That is a real strength. Working against an outside reader's ability to weigh the claim further, the available text gives no author names or institutional affiliation and no names for the open-weight models tested beyond calling them several, so independent verification of the finding, or a sense of which specific systems were tested, is not possible from the text alone.

Risks and caveats

The source text does not say how a user's demographic attributes or expertise were signaled to the models during the experiments, whether through names, explicit prompts, or some other metadata, which limits judging how closely the setup mirrors a real deployment. It also gives no percentages or effect sizes for the reported shifts, so the size of the demographic effect on internal representations cannot be judged from the abstract alone. And because steering vectors are an intervention the authors applied themselves, the paper demonstrates that these internal representations are capable of shaping behavior under that intervention; it does not claim the models act on them unprompted in ordinary use.

“representational biases are often detectable, even when behavioral biases are not visible.”

— the authors