Glonce

LLM-built simple baselines recover most of deep learning's gains on 36 science tasks

researchThe Glonce editorsarXiv🔥60Glonce rating60Impact45Novelty60Relevance90Surprise50Players10Credibility75

A new arXiv paper asks whether large language models can fix a familiar problem in scientific machine learning: missing baselines. To prove the value of a new deep learning method, it has to be compared against appropriate baselines. As deep learning has spread through the physical and life sciences, the authors say, there have been numerous cases where apparent progress turned out to be an artifact of omitted or missing baselines.

The authors tested whether a language model could automate baseline construction. They assembled 36 supervised classification and regression tasks from peer-reviewed studies in Nature, Science and Cell-family journals, each studying a novel deep learning method for a scientific application. For every task, a language model was used to construct a simple baseline.

The results, in the authors' words, are that these automatically constructed baselines are strong. They recovered a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the deep learning model's score. These are shares of that gap, not absolute accuracies. The mean sits below the median, so some tasks were recovered much less well than the typical one. In 12 studies, the median simple baseline matched or exceeded the performance of the deep learning model.

The paper also looks at how results vary across scientific domains and publication venues, and at the effects of test-time compute and of giving the model different levels of study context.

The authors frame the work against the strain on peer review, which they say has struggled to keep up with the rapidly growing volume of research articles. Their results suggest that automated baseline construction is a practical and scalable check on deep learning contributions in the physical and life sciences.

Key facts

  • The study assembled 36 supervised classification and regression tasks from peer-reviewed Nature, Science and Cell-family studies of novel deep learning methods.
  • A language model built a simple baseline for each task; these recovered a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the deep learning model's score.
  • In 12 studies, the median simple baseline matched or exceeded the deep learning model's performance.
  • The paper also examines variation across scientific domains and venues, and the effects of test-time compute and different levels of study context.
  • The authors say the results suggest automated baseline construction is a practical and scalable check on deep learning claims in the physical and life sciences.

Why it matters

Deep learning results in the sciences are only as meaningful as the baselines they beat. The authors point to numerous cases where apparent progress turned out to be an artifact of omitted or missing baselines. This paper tests a cheap remedy: let a language model write the simple baseline. If it recovers a median of 95.5% of the gap on 36 tasks from top journals, the headline gains of many deep learning methods look smaller once a simple comparison exists. In 12 of the 36 studies the median simple baseline matched or exceeded the deep learning model.

Who it affects

Researchers who publish deep learning methods for the physical and life sciences, since their results may be measured against automatically built baselines. Reviewers and journal editors are also affected: the authors argue that human peer review has struggled to keep up with the rapidly increasing volume of research articles, and offer automated baseline construction as a way to help.

How to use it

The abstract presents automated baseline construction as a practical and scalable check on deep learning contributions. The practical reading is that a simple baseline built by a language model can be run alongside a new method before its gains are claimed or accepted. The abstract does not say what kinds of simple baselines were constructed, so the method's details have to come from the full paper.

How solid is it

This is an arXiv preprint, and the claims here come from its abstract. The numbers are specific: 36 tasks, a median of 95.5% and a mean of 85.7% of the gap recovered, and 12 studies where the median baseline matched or exceeded the deep learning model. The authors word their conclusion as results that suggest a practical check, not proof. The abstract names no authors or institutions and does not name which language model was used.

Risks and caveats

A median of 95.5% and a mean of 85.7% mean the typical task was recovered well but some were recovered much less well. Matching the deep learning score in 12 of 36 studies does not say that any original study's conclusions were wrong. The abstract gives no per-task scores or study names, no results on test-time compute or study context, and nothing on which domains or venues varied more. The authors' own wording is that the results suggest the approach is a useful check, not that it replaces review.

“These automatically constructed baselines are strong, recovering a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the score of the deep learning model.”

— arXiv abstract 2610.40700