Glonce

AION-1 relies on detection flags, not pixels, skewing redshift estimates

researchThe Glonce editorsarXiv🔥72Glonce rating72Impact75Novelty75Relevance80Surprise60Players25Credibility85

Researchers ran a causal audit of AION-1, a 39-modality transformer trained on more than 200 million astronomical objects, to test how much of its output genuinely comes from the pixels it is fed. Foundation models for astronomy are trained on both survey pixels and the catalogue products derived from those pixels, and those catalogues are incomplete at a measurable rate; a model trained on both inherits that incompleteness as a systematic error.

The audit intervened on AION-1's inputs directly. Holding the image tokens byte-identical and editing only the survey's detection and segmentation map changed every quantity the model reported, including flux, size, ellipticity and redshift, by 110 to 4400 times a matched placebo edit. The mechanism is detection gating: the model's output correlates with whether a detection is registered at the field's centre (r = 0.47), not with how much light the segmentation mask actually encloses (r = 0.30). Across 322 real blended sources, the model's output showed essentially no relationship to how the processing pipeline had split the light between the blended objects (R = -0.006). The bias is not specific to the detection channel either: feeding the model catalogue photometry that contradicts the image left it nine times worse off than giving it no metadata at all.

The authors then measured how this plays out in a real pipeline. The Legacy Survey pipeline leaves 3.68% of targets with no detection segment covering their position at all. Propagating that miss rate, using the fields the pipeline actually returns to represent each miss, shifted tomographic mean redshift estimates by a median of 0.71 times the LSST DESC requirement across 40 tomographic bin assignments, and exceeded that requirement outright in 12 of the 40. Drawing the missed detections according to their measured dependence on magnitude, rather than uniformly at random, did not change the result. Once observed positional errors were added on top, the worst-affected bin's shift reached 8.3 times the requirement.

Two interventions eliminate the bias: using spectroscopy instead of the detection-derived metadata removes the effect entirely, and simply withholding the detection channel from the model removes it at no measurable cost. The authors also report that the effect grows larger as model scale increases.

The audit traces two further limits to the model's tokeniser. Its image codec resolves only 28 effective states on source patches, against 934 for the spectrum codec, and the redshift readout itself is quantisation-limited. The authors also tested sparse dictionaries as a tool for isolating causal mechanisms inside the model and found them unreliable: across 15 dictionaries, recovery of the true signal ranged from 26% to 75%, and changing only the random seed moved results by up to 18 points.

Key facts

  • Editing only the survey's detection and segmentation map, with image tokens left byte-identical, changes AION-1's reported flux, size, ellipticity and redshift by 110 to 4400 times a matched placebo edit.
  • The bias is detection gating: output correlates with a detection at the field centre (r = 0.47) far more than with the light the mask encloses (r = 0.30); across 322 real blends, correlation with how light was partitioned is essentially zero (R = -0.006).
  • Propagating the Legacy Survey pipeline's 3.68% miss rate shifts tomographic mean redshifts by a median 0.71 times the LSST DESC requirement across 40 assignments, exceeding it in 12, and up to 8.3 times in the worst bin once positional errors are included.
  • Spectroscopy and simply withholding the detection channel both remove the bias at no measurable cost, while the effect grows worse as model scale increases.
  • The tokeniser's image codec resolves only 28 effective states versus 934 for the spectrum codec, and sparse-dictionary causal probes proved unreliable, with recovery spanning 26% to 75% across 15 dictionaries.

Why it matters

Foundation models for astronomy are being built to fuse pixels with catalogue metadata, on the assumption that more inputs mean better inference. This audit shows the opposite can happen silently: AION-1 largely stopped using the pixels and instead keyed its answers to whether the pipeline had flagged a detection at all. Because the model's outputs feed directly into redshift estimates, an error introduced this way is systematic rather than random, and it does not announce itself the way a noisy measurement would.

Who it affects

The direct subject is AION-1 and anyone using it, or a similarly trained multi-modal foundation model, for photometric redshift estimation. The consequence lands on large imaging surveys such as those feeding LSST DESC's tomographic redshift bins, since the bias is driven by exactly the kind of incomplete detection coverage that real pipelines like the Legacy Survey's produce.

How to use it

The audit points to two practical fixes rather than a product to adopt. Supplying spectroscopy instead of detection-derived metadata removes the bias entirely, and withholding the detection channel from the model altogether removes it at no measurable cost to the model's other outputs. Either change can be applied before trusting a foundation model's redshift output in a tomographic analysis.

How solid is it

The core test isolates the mechanism cleanly: image tokens are held byte-identical while only the segmentation map is edited, compared against a matched placebo edit, which is a genuine causal intervention rather than a correlational observation. The result held up across 322 real blended sources and across 40 tomographic bin assignments, and reproducing the miss pattern by magnitude dependence rather than uniformly did not change it. The authors' own limitations section is unusually candid: sparse dictionaries, a common tool for probing model internals, gave recovery ranging from 26% to 75% across 15 dictionaries and shifted by up to 18 points from the random seed alone, which the authors themselves flag as an unreliable causal handle.

Risks and caveats

The material does not name the paper's authors, their institutions, or a publication date, and it does not describe how the matched placebo edit was constructed. It also stops at the redshift bias itself: no specific downstream science result or survey outcome is quantified as having been affected beyond the tomographic bias figures reported here. As a preprint, the work has not been described as peer reviewed in this material.