RelateAnything predicts scene relations from any vocabulary in 20ms

Object detection and segmentation have both become open-vocabulary: a model can be told at inference which classes or regions to look for, rather than being locked to a fixed label set from training. Scene-graph prediction, the task of labeling how objects in an image relate to each other ('person riding horse', 'cup on table'), has not made that jump. Models in this space are still trained and evaluated on a fixed list of 50 or 56 predicates from a single annotation style, with the relation head wired to the object labels of one specific detector. The authors identify three obstacles behind this gap: no existing relation dataset is both free-text and verified for accuracy, a label-conditioned architecture structurally cannot accept a vocabulary it never saw in training, and the standard evaluation metric rewards matching the training corpus, so a model tested on a larger, more open vocabulary scores worse even if it is more capable. The paper presents RelateAnything, a 53M-parameter model that takes an image plus regions from any source and returns scored relations over a predicate vocabulary supplied as plain-text strings at inference time. It never takes object labels as input, so it can be paired with any detector or segmenter without retraining, and its vocabulary is a bank of text embeddings rather than a fixed classifier. It runs at 20 milliseconds per frame. Training a model over the 19,103 predicates the authors use requires positive-unlabeled supervision (since a given image cannot be exhaustively labeled with every true relation) and a text encoder that can tell antonyms apart, which is a problem because standard contrastive text encoders place antonyms at a cosine similarity of 0.95, effectively treating opposite relations as near-identical. To supply that training signal the authors built RA-4M: 474,000 images and 4.3 million relations spanning 10,102 free-text predicates, generated using numbered box markers and verified geometrically. To measure open-vocabulary performance properly, they also built OV-SGG-Bench, a benchmark scoring six axes that standard recall metrics cannot capture. Across three cross-dataset benchmarks plus a fourth zero-shot setting, RelateAnything achieves 2.3 to 3.5 times the mean recall of the strongest open-vocabulary method of comparable scale, margins the authors say hold up even when a real (non-oracle) object detector is used instead of ground-truth regions. It also outperforms a 3-billion-parameter vision-language model built for scene-graph generation on both metrics, while using under 2% of that model's parameters. The authors additionally report that in-domain measurement, testing on data similar to training, overstates how well such models transfer to new domains by about 5 times. The model, the RA-4M corpus, and the OV-SGG-Bench benchmark are all released publicly.

Key facts

RelateAnything is a 53M-parameter model that predicts relations between image regions using any predicate vocabulary given as free text at inference, running at 20 ms/frame.
It never takes object labels as input, decoupling relation prediction from any specific detector.
The authors built RA-4M (474k images, 4.3M relations, 10,102 free-text predicates) and OV-SGG-Bench, a six-axis benchmark, and released both publicly along with the model.
Across three cross-dataset benchmarks and a fourth zero-shot test, it reaches 2.3 to 3.5x the mean recall of the strongest comparable open-vocabulary method, and beats a 3B-parameter VLM scene-graph model using under 2% of its parameters.
The authors report in-domain evaluation overstates real transfer performance by about 5x, and that standard contrastive text encoders confuse antonyms at 0.95 cosine similarity, a problem their training approach had to correct for.

Why it matters

Detection and segmentation left fixed label sets behind years ago; scene-graph prediction, which describes how objects relate rather than just what they are, has stayed locked to whichever 50-to-56-word predicate list its training set used. That means a scene-graph model trained on one dataset's vocabulary cannot be pointed at a new domain without retraining, and worse, the standard recall metric actively penalizes models tested on a broader vocabulary than they were trained on, since it rewards matching the training corpus rather than being right. RelateAnything is presented as removing the label-conditioning that causes this: it takes no object labels as input at all, only regions and a text vocabulary, so the same model works with any upstream detector and any predicate list handed to it at inference.

Who it affects

This is infrastructure for anyone building systems that need to understand relationships between objects in images or video, not just what the objects are: robotics and scene understanding, visual question answering, image captioning pipelines, and any application layering reasoning on top of open-vocabulary detectors and segmenters. Because the authors release the model, the RA-4M training corpus and the OV-SGG-Bench benchmark publicly, both practitioners who need a drop-in relation predictor and researchers building or evaluating competing scene-graph methods are directly affected.

How to use it

RelateAnything is designed as a modular component: feed it an image plus regions from any source (any detector or segmenter), plus a vocabulary of relation predicates as plain strings, and it returns scored relations without needing to know the object class labels those regions came from. At 20 ms per frame it targets real-time use. The paper states the model, RA-4M and OV-SGG-Bench are all public, though no license, hosting location, or release date beyond that statement is given in the source.

How solid is it

The evaluation spans three cross-dataset benchmarks plus a separate zero-shot benchmark, and the authors report the recall margins over the strongest comparable open-vocabulary method (2.3 to 3.5x) hold when a real, non-oracle detector is substituted for ground-truth regions, which is a meaningful robustness check since many scene-graph results only hold with oracle inputs. The comparison against a 3B-parameter VLM scene-graph baseline, where RelateAnything wins on both metrics using under 2% of the parameters, is a strong efficiency claim. The authors also built their own benchmark (OV-SGG-Bench) partly because they argue the standard metric is structurally unfair to open-vocabulary models, which is a reasonable methodological point but also means part of the evaluation is on a benchmark the same team designed.

Risks and caveats

The source text is the paper abstract; it names no authors, institutions, publication venue or date, and does not identify the specific 'strongest open-vocabulary method' or '3B-VLM scene-graph model' RelateAnything is compared against, so the exact baselines cannot be checked from this text alone. The authors' own finding that in-domain measurement overstates transfer gains by roughly 5x is itself a caution about how scene-graph results are usually reported, and by extension applies some pressure back onto their own cross-dataset numbers, which the paper addresses by testing zero-shot and with a real detector, but readers should still treat the headline recall multiples as measured under the paper's own benchmark design.