Glonce

Small models fine-tuned on Psych-101 match a 70B baseline in-distribution

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact60Novelty75Relevance95Surprise57Players25Credibility80
Small models fine-tuned on Psych-101 match a 70B baseline in-distribution

A team of researchers fine-tuned fourteen language models, ranging from 135M to 14B parameters and spanning four architecture families, on Psych-101, a dataset of 10.7 million trial-level human choices drawn from 160 psychological experiments. The goal was to test how much scale this kind of cognitive fine-tuning actually needs, and whether the resulting models learn genuine task structure or simply exploit statistical shortcuts in the training data.

On held-out participants drawn from the same experiments the models were trained on, scale barely mattered: models across the full parameter range fell within a narrow performance band, and models with just 0.6B to 1B parameters matched a 70B baseline. Outside that distribution, on novel task structures the models had not seen during training, that narrow band opened into a much steeper scaling gradient, with larger models clearly ahead at generalizing to the new tasks.

To find out what information the models were actually using, the researchers ran two diagnostics across 27 of the experiments. First, they progressively stripped four separate prompt channels, task instructions, experimental stimuli, outcome feedback, and choice history, to see which ones performance depended on. Masking the content of stimuli and feedback destroyed 75.7% of the models' learned information and pushed their performance below chance, which shows that choice history alone cannot account for how the models perform. Second, they permuted the order in which trials were presented. This left performance unchanged on tasks where each trial is independent, but hurt performance on tasks where trial order is shaped by a participant's prior responses, indicating the models pick up on that structure when it is present.

The researchers conclude that small, cognitively fine-tuned models show promise as noise ceiling estimators for psychological experiments, though their scope remains bounded by the paradigms seen in training.

Key facts

  • Fourteen language models, from 135M to 14B parameters across four architecture families, were fine-tuned on Psych-101, a dataset of 10.7 million trial-level human choices from 160 experiments.
  • In-distribution, on held-out participants, scale barely mattered: models with just 0.6B to 1B parameters matched a 70B baseline.
  • Out-of-distribution, on novel task structures, that narrow performance band opened into a much steeper scaling gradient, with larger models clearly ahead at generalizing.
  • Stripping the content of stimuli and feedback across 27 experiments destroyed 75.7% of the models' learned information and pushed performance below chance, showing choice history alone does not explain their performance.
  • Permuting trial order left performance unchanged on tasks with independent trials but hurt performance where trial order depends on prior responses; the authors conclude small fine-tuned models could serve as noise ceiling estimators, bounded by the paradigms seen in training.

Why it matters

Large language models fine-tuned on human behavioral data have emerged as general-purpose cognitive proxies, standing in for human participants in psychological research, but how much scale that actually requires, and whether these models process real task structure or exploit statistical shortcuts, have been open questions. This study answers the scale question directly for in-distribution use: performance barely improves past 0.6B to 1B parameters, and a model at that size matches a 70B baseline. That carries a direct efficiency implication, since a much smaller and cheaper model reproduces most of what a far larger one can do, at least within the experiments and participants the model has already been trained on.

Who it affects

The direct audience is researchers using language models as proxies for human decision-making in psychology and behavioral science, plus anyone deciding how much compute to spend fine-tuning a model for that purpose. For work confined to the kinds of tasks and participants already represented in a training set like Psych-101, a small model in the 0.6B to 1B range is shown to do the job. For work that needs to generalize to novel task structures the model has not seen, the study found larger models clearly ahead, so the right model size depends heavily on how in-distribution the intended use case is.

How to use it

The models were fine-tuned on Psych-101, an existing dataset of 10.7 million trial-level human choices from 160 experiments. The diagnostic method described, progressively stripping prompt channels (task instructions, experimental stimuli, outcome feedback, choice history) and permuting trial order across a set of experiments, offers a way to test whether a fine-tuned model has learned genuine task structure or is relying on shortcuts, and researchers building or evaluating similar cognitive models could apply it to their own setups. The source text does not give the specific names of the fourteen models or four architecture families used, nor any licensing or release details, so implementation specifics beyond the described method are not available here.

How solid is it

The central claims rest on controlled ablations rather than a single aggregate score. Stripping the content of stimuli and feedback across 27 experiments measurably destroyed 75.7% of learned information and pushed performance below chance, which rules out choice history alone as the source of the models' performance. A separate trial-order permutation test adds an independent check: it left performance unchanged on tasks where trials are independent, and hurt performance specifically on tasks where trial order depends on prior responses. That the models are sensitive to order only where order carries real information, and invariant where it does not, supports the paper's reading that they are picking up genuine task structure rather than a generic shortcut.

Risks and caveats

The study's own out-of-distribution results are its clearest caveat: the narrow in-distribution performance band, where small models match a 70B baseline, opens into a much steeper scaling gradient once task structure is novel, so the headline finding does not extend to generalization. The authors state the models' scope remains bounded by the paradigms seen in training. The source text also does not name the study's authors or institutions, nor identify the four architecture families or the fourteen specific models used, which limits independent verification of the results.