Glonce

New benchmark KoNA tests whether vision-language models know when to say no

researchThe Glonce editorsHugging Face🔥69Glonce rating69Impact62Novelty72Relevance95Surprise42Players28Credibility82
New benchmark KoNA tests whether vision-language models know when to say no

Researchers have introduced KoNA, a benchmark built to test a specific failure mode in vision-language models (VLMs): what happens when a single request mixes something the model can legitimately answer with something it should refuse, correct, or decline. Most existing benchmarks treat non-compliance as an all-or-nothing property of the whole query, on the assumption that a request either deserves a full answer or deserves a full refusal. KoNA argues that real-world queries do not sort so cleanly: a user might ask one compound question where part rests on a false premise, part asks about something not visible in the image, or part is unsafe, while the rest of the question is perfectly answerable. The benchmark spans five categories of problematic content: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. For each category, KoNA evaluates two distinct capabilities: whether a model withholds compliance when an entire query falls into one of these categories (query-level non-compliance), and whether it can withhold compliance on just the problematic component while still answering the answerable part, using paired single and compound queries to test this directly. The authors ran an evaluation across a range of VLMs and found that models frequently fail to refuse, correct, or abstain in the appropriate way, and that this failure gets substantially worse specifically on the compound queries that require selective, component-level non-compliance rather than an all-or-nothing response. To address the gap, the authors fine-tuned VLMs on KoNA examples that call for selective non-compliance, mixed with a set of fully answerable examples that should simply be answered directly. The fine-tuned models showed substantial improvements in non-compliance accuracy while largely preserving performance on the fully answerable tasks, which the authors take as evidence that the models learned to distinguish which parts of a compound query can be answered and which cannot, and to respond accordingly rather than defaulting to either blanket compliance or blanket refusal.

Key facts

  • KoNA is a new benchmark for selective non-compliance in vision-language models: refusing or hedging on only the problematic part of a mixed request rather than the whole query.
  • It covers five categories of problematic content: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety.
  • Each task tests two capabilities, query-level and component-level non-compliance, using paired single and compound queries.
  • Evaluation across diverse VLMs found models often fail to refuse, correct, or abstain appropriately, and failures grow more pronounced on compound queries needing selective non-compliance.
  • Fine-tuning on KoNA examples plus fully answerable data substantially improved non-compliance accuracy while largely preserving performance on fully answerable tasks.

Why it matters

Most non-compliance benchmarks score a VLM's response to a whole query as either compliant or refused, which hides a common real-world case: a single request that bundles an answerable part with a part that rests on a false premise, references something not visible in the image, asks about something unknowable, is infeasible, or is unsafe. KoNA is built specifically to catch models that handle the all-or-nothing case reasonably well but collapse when a query demands answering part and declining part in the same response.

Who it affects

The benchmark targets developers and researchers building or evaluating vision-language models, particularly anyone deploying VLMs in settings where user queries are compound or ambiguous rather than cleanly answerable or cleanly out of scope, such as visual assistants and multimodal chat systems.

How to use it

KoNA is structured as five categories, each testing both query-level non-compliance (the whole request should be declined) and component-level non-compliance (only part of a compound request should be declined), via paired single and compound query sets. The authors also demonstrate a fine-tuning recipe: training on KoNA examples requiring selective non-compliance together with a fully answerable set, which improved non-compliance accuracy without much loss on tasks that should simply be answered.

How solid is it

The claims come from the paper's own abstract: an evaluation across multiple VLMs establishes the baseline failure pattern, and the same authors report the fine-tuning results. No specific model names, dataset size, or numeric accuracy figures for either the baseline or fine-tuned results are given in the available text, so the magnitude of the reported improvements cannot be independently assessed from this description alone.

Risks and caveats

The source text does not name the specific VLMs evaluated or fine-tuned, does not quantify the baseline failure rate or the fine-tuned improvement with numbers, and does not state the size of the KoNA dataset or a release venue and date. The description of the benchmark and results should be read as the authors' own characterization pending release of the full paper and any independent evaluation.