VSysBench finds system messages hurt multimodal LLM accuracy

Production deployments of multimodal large language models (MLLMs) increasingly rely on system messages, the instructions set behind the scenes to govern how a model behaves, separate from whatever a user types directly. But existing benchmarks for checking whether models actually keep those instructions test constraints only in text, or embed them into the user's own turn instead of the system message, leaving system-message adherence in multimodal settings largely unmeasured. They also leave open a separate question: whether following those constraints costs a model some of its underlying vision-language capability.

To close that gap, the authors introduce VSysBench, a benchmark built on the existing MMVet-v2 vision-language benchmark. It organizes system-message constraints into 5 main categories and 22 sub-categories, ranging from purely textual directives applied in a visual context to constraints that are fully grounded in the image itself. Each category is paired with a misaligned counterpart, a version where a conflicting user instruction is added, which stress-tests whether the model still follows the instructional hierarchy and keeps prioritizing the system message. Every response is scored jointly on two axes at once, constraint compliance and answer correctness, through two metrics: the Joint Satisfaction Rate (JSR) and the Cross-Constraint Sensitivity (CCS).

Testing 16 MLLMs on VSysBench, the authors find that imposing system messages substantially erodes base task accuracy across the board. When a user instruction conflicts with the system message, compliance collapses for open-weight models, while it stays stable for the top proprietary ones. Across every model tested, vision-grounded constraints, the ones requiring the model to act on what is actually in the image rather than just follow a textual rule, proved the hardest category to satisfy.

Key facts

Built on MMVet-v2, VSysBench organizes system-message constraints into 5 main categories and 22 sub-categories, spanning textual directives in visual contexts to fully vision-grounded ones.
Each category is paired with a misaligned counterpart: a version with a conflicting user instruction that stress-tests whether the model still follows the system message over the user turn.
Responses are scored jointly on compliance and correctness through two metrics, the Joint Satisfaction Rate (JSR) and the Cross-Constraint Sensitivity (CCS).
Across 16 evaluated MLLMs, imposing system messages substantially erodes base task accuracy.
Under conflicting user instructions, compliance collapses for open-weight models but stays stable for top proprietary ones, and vision-grounded constraints are the hardest category for every model.

Why it matters

System messages are how production MLLM deployments set behavior boundaries behind the scenes, separate from whatever a user types. Before VSysBench, benchmarks for checking whether models keep those boundaries tested constraints only in plain text or folded them into the user's own turn, so nobody had measured system-message adherence in a genuinely multimodal setting, or whether holding to a constraint costs the model some of its underlying vision-language accuracy. VSysBench answers both questions at once, and the answer is that the tradeoff is real: across the 16 models tested, imposing a system message substantially erodes base task accuracy.

Who it affects

Anyone building a product on top of an MLLM that leans on a system message to hold a behavior boundary in place, such as a content policy, a fixed persona, or a required output format, and needs that boundary to survive a user pushing back on it. The paper's split between open-weight and proprietary models under conflicting user instructions bears directly on model choice for that kind of deployment.

How to use it

VSysBench is a benchmark, not a shipped tool with a price or license. Its practical use is as a check before deploying a model into a system-message-reliant role: work through its five constraint categories and 22 sub-categories, from textual directives up to fully vision-grounded ones, and see how a candidate model scores on the paper's two joint metrics, especially on the misaligned, conflicting-instruction version of each test, since that is where the paper reports the sharpest difference between open-weight and proprietary models.

How solid is it

VSysBench builds on an existing vision-language benchmark, MMVet-v2, rather than starting from nothing, and adds a taxonomy of 5 categories and 22 sub-categories, each paired with a misaligned counterpart, instead of testing compliance in isolation. It scores each response on two axes together, compliance and correctness, through the JSR and CCS metrics, and the reported findings come from testing 16 different MLLMs rather than one or two. The abstract does not name the paper's authors or their institutions, and it does not give the formulas behind JSR or CCS, so the exact scoring mechanics cannot be checked from this text alone.

Risks and caveats

The abstract does not name any of the 16 evaluated MLLMs, nor does it say which of them count as open-weight versus proprietary, so the reported split between the two groups cannot be matched to specific models from this text. It also gives no numeric size for how much accuracy system messages erode, and no explanation for why vision-grounded constraints are harder than textual ones for every model tested.