AltRA-Test extends Alt-Test to judge when LLMs can replace human annotators

AltRA-Test extends Alt-Test to judge when LLMs can replace human annotators

LLMs are increasingly used as judges and graders in place of human annotators, which raises a question: when is that replacement statistically justified? An earlier method, the Alternative Annotator Test (Alt-Test), was built for dense settings in which every item is annotated by the same annotators. In many large-scale annotation projects that is not the case, because items are annotated by different subsets of annotators.

The paper proposes the Alternative Annotator Test with Resampled Annotations (AltRA-Test) for exactly that situation. It does not require all items to be annotated by the same annotators. The procedure is a bootstrap. It resamples items, draws k human annotations for each item, and replaces one of those annotations with an LLM annotation. The result is a pair of panels: an all-human panel and a mixed human-LLM panel. A replacement is supported when the inter-annotator agreement (IAA) of the mixed panel is statistically non-inferior to the IAA of the all-human panel. In other words, the LLM passes when it preserves the signal that the human annotators share. The authors establish that AltRA-Test is a consistent test of asymptotic level alpha.

The evaluation has two parts. In simulations, where it is known whether the LLM is a better or a worse annotator than the humans, AltRA-Test makes more correct replacement decisions than existing baselines. In real-world experiments, the authors test 18 LLMs and 5 prompting techniques on 20 datasets spanning text, image, audio and video. LLMs pass the test in about half of the setups, depending mainly on the task. Across all these evaluations, about 100 items often suffice for the test.

The authors' conclusion is a cautious one: since replacement is not always justified, they encourage researchers to test it for their own task.

Key facts

  • AltRA-Test (Alternative Annotator Test with Resampled Annotations) extends the earlier Alt-Test to settings where items are annotated by different subsets of annotators.
  • It uses a bootstrap that draws k human annotations per item and swaps one for an LLM annotation, then checks whether the mixed panel's inter-annotator agreement is statistically non-inferior to the all-human panel's.
  • The authors establish that the test is consistent and of asymptotic level alpha; in simulations it makes more correct replacement decisions than existing baselines.
  • Real-world experiments cover 18 LLMs and 5 prompting techniques on 20 datasets across text, image, audio and video; LLMs pass in about half of the setups, depending mainly on the task.
  • About 100 items often suffice for the test.

Why it matters

Using an LLM as a judge or grader in place of human annotators is becoming common, but the decision to do so often lacks a statistical basis. The earlier Alt-Test gave such a basis only when all items were labelled by the same annotators. Large annotation projects often do not work that way, so AltRA-Test covers a case the earlier test did not. The finding that LLMs pass in only about half of the real-world setups is a reminder that replacement cannot be assumed.

Who it affects

Researchers and teams who collect human labels or ratings and are considering an LLM as a cheaper substitute, especially where items were annotated by different subsets of annotators. The tested datasets span text, image, audio and video, so the setting is not limited to one modality.

How to use it

The idea is to run the test on your own task before swapping humans for an LLM. You need human annotations for a sample of items, plus the LLM's annotations on the same items. The test then compares agreement among all-human panels with agreement in panels where one human is replaced by the LLM. According to the authors, about 100 items often suffice. No code or data release is mentioned.

How solid is it

The authors back the test with a result that it is consistent and of asymptotic level alpha, and with both simulations and real-world data. In simulations, where the truth about the LLM is known, it makes more correct replacement decisions than existing baselines. This is an HF-listed preprint abstract: the baselines are not named, and no effect sizes are given. The claim that about 100 items suffice is stated as 'often', not always.

Risks and caveats

Passing the test depends mainly on the task, so a result for one task does not carry over to another. The roughly 100-item figure is a general observation, not a guarantee. Names of the 18 LLMs, the 5 prompting techniques and the 20 datasets are not listed, and the values of k and alpha are not stated, so the scope of the real-world results is hard to judge from the abstract. As the authors put it, replacement is not always justified.

“Since replacement is not always justified, we encourage researchers to test it for their task.”

— AltRA-Test paper abstract