Glonce

LOTUS pretrains time-series language models to forecast the next patch

researchThe Glonce editorsarXiv🔥56Glonce rating56Impact40Novelty60Relevance95Surprise25Players10Credibility75

Time-series language models usually align numerical signals with language through labels, captions or question-answer pairs. The paper argues that this supervision encourages linguistic representations but does not explicitly require them to preserve information about how the underlying signal evolves.

The authors propose forecast grounding, a pretraining principle: language-aligned time-series representations must remain predictive of the future evolution of the signal before they are adapted to downstream tasks.

They instantiate the idea in LOTUS. The model divides each time-series channel into non-overlapping patches and maps those patches into the embedding space of a large language model. It then predicts the next patch from the LLM hidden states of the time-series tokens, which pushes the learned representation to retain temporal information. A second contribution is a temporal-spectral patch representation: each raw patch is combined with its spectral representation before being mapped into the embedding space, so the model can capture both temporal changes and periodic patterns.

Across three language model backbones and six downstream evaluation datasets, LOTUS reaches the best average performance with minimal trainable parameters. It reaches 59.91% accuracy and outperforms the strongest matched baseline by 3.36%. Representation analyses further show that LOTUS uses the numerical input and preserves temporal structure in its learned patch representations. The authors conclude that the results establish forecast grounding as a promising pretraining principle, one that lets compact time-series language models outperform models 25 to 50 times their size on TSRBench.

Key facts

  • Forecast grounding is a pretraining principle: language-aligned time-series representations must stay predictive of the signal's future evolution before downstream adaptation.
  • LOTUS splits each channel into non-overlapping patches, maps them into the LLM embedding space and predicts the next patch from the LLM hidden states of the time-series tokens.
  • A temporal-spectral patch representation pairs each raw patch with its spectral representation to capture temporal changes and periodic patterns.
  • Across three backbones and six downstream datasets, LOTUS reports the best average performance: 59.91% accuracy, 3.36% above the strongest matched baseline.
  • Compact LOTUS-style models are reported to outperform models 25 to 50 times their size on TSRBench.

Why it matters

Models that read numerical signals through a language model are usually trained to answer questions or produce captions about a series. This paper points at a gap in that recipe: nothing forces the model's representation to keep track of how the signal actually evolves. Forecast grounding fills the gap with a simple objective, predicting the next patch, applied before any downstream adaptation. If the reported gains hold, it suggests that a forecasting signal can matter more than raw model size for time-series understanding, since compact models are reported to beat models 25 to 50 times larger on TSRBench.

Who it affects

Researchers building time-series language models and benchmarks such as TSRBench are the direct audience. Teams that want to put a language interface on sensor, financial or other numerical data streams may care about the pretraining idea, since the paper claims strong results with minimal trainable parameters.

How to use it

The practical takeaway is the recipe. Cut each channel into non-overlapping patches, combine each raw patch with its spectral representation, map the result into the LLM embedding space, and train the model to predict the next patch from the hidden states of the time-series tokens before adapting it to downstream tasks. No code, dataset or model release is mentioned in the source, so reproducing this means implementing it from the description.

How solid is it

This is a preprint abstract, and every claim comes from the authors themselves. The headline numbers are specific: 59.91% accuracy as the best average across three language model backbones and six downstream datasets, and a 3.36% margin over the strongest matched baseline. The source does not say whether that 3.36% is in percentage points or relative terms. It also does not name the three backbones, the six datasets or the strongest baseline, and it does not give the parameter count of LOTUS or of the larger models it outperforms. The 59.91% is stated as average performance across the six downstream datasets, not as a TSRBench score.

Risks and caveats

Without the backbone names, dataset list and baseline identity, the size of the improvement is hard to judge. The comparison with models 25 to 50 times larger is reported on TSRBench only. The representation analyses are the authors' own evidence that the model uses the numerical input and keeps temporal structure. Independent replication has not been reported in the source.