No-persona LLM baseline beats synthetic personas at predicting clicks
Marketers increasingly ask large language models to role-play as "synthetic personas" to predict how an audience will react to a piece of copy before it ships, on the theory that a profile-conditioned model mimics a human sample closely enough to be useful. A new study tests that premise against real behaviour rather than against other model outputs. The researchers used the Upworthy Research Archive, a set of thousands of headline A/B tests run on shared real traffic with measured click-through, as held-out ground truth. They compared a ten-persona panel, built from the real audience's demographics, against a no-persona zero-shot baseline that just asks the model how likely a typical reader is to click each headline variant. Two findings stand out. First, the ground truth itself is noisy: most of the Upworthy A/B tests have no statistically distinguishable winner, so predictive validity could only be measured on a reliable subset of 399 tests where a winner is clear. Second, and running against the premise the whole approach is built on, adding personas made predictions worse, not better. On the reliable subset, the no-persona baseline ranked headline variants with a Kendall tau of 0.361 (a medium effect) and picked the actual winning variant 49.2% of the time. The ten-persona panel managed only a Kendall tau of 0.084 and 34.6% top-1 accuracy, with confidence intervals for the two conditions that do not overlap. The authors read this as evidence that asking a model directly taps an accurate population-level prior, while forcing it to role-play specific personas injects bias and noise instead of removing it. The result held up under stress-testing: it replicates across three independent splits of the Upworthy data, holds in the same direction on a separate news dataset from a different domain, and is robust to changes in random seed, prompt phrasing and choice of model, including three tiers of Gemini and a different model family, OpenAI's gpt-4.1, where the paired gap between the two conditions was also significant. The authors' conclusion is blunt: for predicting aggregate engagement, a plain LLM ranker beats persona simulation, and synthetic personas are not merely a weak predictor but worse than not using them at all. All the numbers in the paper are said to regenerate from a public, artifact-first replication package.
Key facts
- No-persona baseline: Kendall tau 0.361, top-1 accuracy 49.2% at picking the winning headline variant
- Ten-persona panel, grounded in real audience demographics: Kendall tau 0.084, top-1 accuracy 34.6%, with non-overlapping confidence intervals versus the baseline
- Ground truth came from the Upworthy Research Archive; predictive validity was measured on a reliable subset of 399 A/B tests with a statistically distinguishable winner
- Result replicates across three independent Upworthy splits, holds in direction on a separate news dataset, and is robust across three Gemini tiers plus OpenAI's gpt-4.1
- All reported numbers are said to regenerate from a public, artifact-first replication package
Why it matters
Persona-conditioned LLMs are already a standard pitch in ad tech and marketing research: run copy past a simulated panel before it ships, and skip or shrink real audience testing. That pitch rests on the assumption that a model told to play a persona predicts real behaviour at least as well as a plain model asked directly. This study tests that assumption against actual click-through data rather than against other synthetic outputs, and finds the opposite: the persona machinery does not just fail to help, it actively hurts prediction accuracy compared with simply asking the model.
Who it affects
Anyone using or selling LLM-based audience simulation to pretest headlines, ads or other copy before launch, and anyone building or evaluating such tools for that purpose. The result speaks specifically to predicting aggregate click-through and ranking copy variants; the paper does not claim to settle every other use of persona simulation.
How to use it
The practical takeaway for a team deciding how to prompt a model for this kind of prediction is to skip persona role-play and ask directly how a typical reader would respond. The authors say the full set of numbers in the paper regenerates from a public, artifact-first replication package, so the comparison can be rerun rather than taken on faith.
How solid is it
The comparison is grounded in real behavioural data (measured click-through on live traffic) rather than another model's judgment, which is what makes it a sim-to-real test rather than a sim-to-sim one. The authors first isolated a reliable subset of 399 A/B tests with a statistically distinguishable winner, since most tests in the archive did not have one, and measured both approaches only on that subset. The gap between the no-persona baseline and the persona panel held across three independent splits of the Upworthy data, in the same direction on a separate news dataset, and across changes in random seed, prompt phrasing and model, including three Gemini tiers and OpenAI's gpt-4.1, where the paired gap was also statistically significant.
Risks and caveats
Most A/B tests in the Upworthy archive had no statistically distinguishable winner at all, which is why validity could only be checked on the smaller 399-test reliable subset rather than the full archive. For the gpt-4.1 replication, the paper reports only that the paired gap was significant, without giving the same Kendall tau or top-1 accuracy figures quoted for the main results. The findings are specific to predicting aggregate engagement and ranking copy variants against real click-through; they do not by themselves rule out other roles for persona simulation.