Glonce

RA-DPO reaches full-data DPO performance in sexism detection using just 30% of pairs

researchThe Glonce editorsarXiv🔥68Glonce rating68Impact55Novelty65Relevance100Surprise45Players45Credibility80

Sexism detection online remains an open problem, and the authors argue existing systems oversimplify it. Because sexism detection is inherently subjective, human annotators often disagree on whether a given post counts as sexist, yet most existing detection systems collapse that disagreement into a single majority-vote label and then treat every instance the same way afterward. The authors say this throws away two signals that carry real information: how much the annotators agreed with each other, and how confident the model itself is in its own prediction.

To use those signals instead of discarding them, the authors propose RA-DPO, Reliability-Aware Direct Preference Optimization. RA-DPO builds a single reliability score for each training example out of three inputs: annotator agreement, the model's own confidence, and a token-level uncertainty signal. That score does two jobs. During training, it selects the highest-value preference pairs to train on with DPO, a preference-based fine-tuning method. At inference time, it supports abstention: the deployed model can decline to answer on inputs it is not confident about, trading coverage, how many inputs it answers, for accuracy on the ones it does.

The authors evaluate RA-DPO on 6,920 multilingual posts from EXIST 2023. They fine-tune OpenAI's gpt-4o base model with RA-DPO via DPO, and separately validate the approach on two open-weight 3B models, Llama and Qwen. The text does not name the specific languages in the post set beyond calling it multilingual, and gives no author names, institutional affiliation, or publication date.

On the training side, the authors report that training on only the top 30% most reliable preference pairs, as ranked by the reliability score, matches the performance of standard DPO trained on the full dataset. They take that as evidence that reliability-aware data selection can reduce training cost without sacrificing performance, though the text does not put a number on how much time, compute, or money that saves.

On the inference side, letting the model abstain rather than force a call, what the paper calls selective prediction, reaches 96.2% accuracy at 50% coverage in what the authors call the true-agreement setting, and 88.7% at the same 50% coverage in what they call the deployable predicted-agreement setting. Both beat an 85.3% baseline that ignores agreement information entirely. At 50% coverage the model is issuing a call on only half of the inputs and abstaining on the rest; the text does not state which of the three trained models, the gpt-4o fine-tune, Llama, or Qwen, these particular figures were measured on.

The authors' overall conclusion is that accounting for annotation uncertainty is beneficial for both efficient training and reliable deployment in subjective classification generally, not only sexism detection.

Key facts

  • RA-DPO (Reliability-Aware Direct Preference Optimization) combines annotator agreement, model confidence, and a token-level uncertainty signal into one reliability score, used to select high-value preference pairs during DPO training and to support inference-time abstention.
  • Evaluated on 6,920 multilingual posts from EXIST 2023, RA-DPO fine-tunes OpenAI's gpt-4o base model via DPO and is separately validated on two open-weight 3B models, Llama and Qwen.
  • Training on just the top 30% most reliable preference pairs matches full-data DPO performance, which the authors take as evidence that reliability-aware selection can cut training cost without sacrificing performance.
  • At 50% coverage, selective prediction, letting the model abstain on uncertain inputs, reaches 96.2% accuracy in the true-agreement setting and 88.7% in the deployable predicted-agreement setting, both above an 85.3% no-agreement baseline.
  • The authors conclude that accounting for annotation uncertainty benefits both efficient training and reliable deployment across subjective classification tasks generally, not just sexism detection.

Why it matters

Sexism detection is inherently subjective: human annotators often disagree on whether a post crosses the line, but most detection systems throw that disagreement away, reducing it to one majority-vote label and treating every case the same afterward regardless of how contested it was. RA-DPO instead folds annotator agreement together with the model's own confidence and a token-level uncertainty signal into a single reliability score, and puts that score to work twice: choosing which training pairs are worth training on, and letting the deployed model decline to answer when it is unsure. That second use, inference-time abstention, is what lets a subjective classifier trade coverage for accuracy instead of being forced to guess on every input.

Who it affects

Anyone building or maintaining a classifier for a subjective, contested category: the paper's own case is online sexism, where human annotators routinely disagree on the right label. The training-time technique speaks to teams who want to cut annotation and training cost without hurting accuracy; the inference-time abstention speaks to teams deploying a classifier where a wrong automated call is costly, since the model can decline rather than force a guess. The authors argue the underlying idea, accounting for annotation uncertainty, is not specific to sexism and extends to subjective classification more broadly, though the paper's own evaluation is on the EXIST 2023 dataset.

How to use it

This is a training and evaluation method, not a released tool or product, so there is no price or license to note. The recipe as described: build a reliability score per training example from annotator agreement, model confidence, and a token-level uncertainty signal; use that score to select the most reliable preference pairs (the paper reports the top 30% is enough) for DPO fine-tuning instead of the whole dataset; and at inference, use the same kind of signal to let the model abstain on inputs it is unsure about instead of forcing an answer, at whatever coverage level fits the deployment. The level tested here is 50%. In the paper, the base model fine-tuned this way is OpenAI's gpt-4o, with the approach separately validated on two open-weight 3B models, Llama and Qwen.

How solid is it

The evaluation runs on 6,920 multilingual posts from EXIST 2023, with RA-DPO fine-tuning OpenAI's gpt-4o base model via DPO and separate validation on two open-weight 3B models, Llama and Qwen. OpenAI appears in the text only as the maker of the gpt-4o base model that gets fine-tuned; the text does not say OpenAI took part in the research itself. The training-side result, that the top 30% most reliable pairs matches full-data DPO, is stated as a performance match rather than a specific accuracy number. The inference-side result is quantified: 96.2% accuracy at 50% coverage in the true-agreement setting and 88.7% at the same coverage in the deployable predicted-agreement setting, both above an 85.3% no-agreement baseline. Two gaps limit how far those numbers can be checked from the text alone: it does not say which of the three trained models these particular accuracy figures were measured on, and it does not define what separates the true-agreement setting from the predicted-agreement setting. The text also gives no author names, institutional affiliation, or publication date.

Risks and caveats

The 50% coverage figures mean the model is issuing a call on only half of the inputs in that setting and abstaining on the rest, so the accuracy gain comes bundled with a real drop in how many cases get an automatic decision at all; a deployment still needs a plan for the abstained half. The reported training-cost saving from using the top 30% of pairs is not quantified in time, compute, or money, only as matching full-data performance. The post set is described only as multilingual, with no languages named, and the gap between the true-agreement and predicted-agreement settings is not defined in the text, which limits how precisely the deployable, predicted-agreement number can be interpreted.

“Sexism detection is inherently subjective, yet most existing systems reduce multi-annotator labels to a single majority decision and treat all instances uniformly.”

— the authors