Glonce

Paper measures GPT-4o mini's instruction capacity at about 37-38 rules

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact40Novelty60Relevance95Surprise30Players15Credibility72
Paper measures GPT-4o mini's instruction capacity at about 37-38 rules

The paper starts from a gap it describes: language models follow instructions less well as the load grows, yet there is still no quantity that expresses a model's limit in units of load, with an uncertainty attached.

To fill it, the authors propose a measurement instrument. The load is a set of rules that each replace a single-token word in a text that carries no meaning. According to the authors, this makes every instruction atomic, the rules independent by the construction of the task, and the outcome of each rule application determined by a computable reference answer. Every occurrence of a word to be replaced counts as a Bernoulli trial, with a known position in the answer and a known rule number.

A statistical model then works in steps. It first measures how the probability of a right replacement depends on the text length, the word position, the replacement density, the number of rules and the rule number. It then describes that dependence with smooth laws and restores a reference level that does not depend on the choice of the coding. The output is the instruction capacity: the number of rules the model holds at a given share of right replacements, reported with an uncertainty.

The authors applied the instrument to GPT-4o mini, using 2958 requests and 642,430 trials. The measurement gave regular laws for every load factor and for the collapse of the answer. The instruction capacity came out at about 37-38 rules at a share of right replacements of 0.95, and at about 18-19 rules at the full replacement density.

The code, the measurement data and the model's "passport" of results are open, and every number in the passport is recomputed from the published points. The authors say the passport of a model serves as the calibration of a future meter of the instruction load of prompts.

Key facts

  • The paper defines instruction capacity as the number of rules a model holds at a given share of right replacements, reported with an uncertainty.
  • The load is made of rules that replace single-token words in a text without meaning, so each rule is atomic and independent, and each outcome is checked against a computable reference answer.
  • GPT-4o mini was measured with 2958 requests and 642,430 trials (individual word replacements treated as Bernoulli trials).
  • Result for GPT-4o mini: about 37-38 rules at a 0.95 share of right replacements, and about 18-19 rules at the full replacement density.
  • Code, measurement data and the passport are open, and every passport number is recomputed from the published points.

Why it matters

The authors point to a missing piece in how instruction-following is discussed: quality is known to fall as the load grows, but no quantity states a model's limit in units of load with an uncertainty. Instruction capacity is their proposal for that quantity. Because the rules are atomic and independent and every replacement can be scored against a computable answer, each occurrence becomes a clean statistical trial, which is what lets the limit come with an error estimate.

Who it affects

The work is aimed at people who want to compare or characterise how many simultaneous instructions a model can carry, since it gives them a number in rules rather than a vague sense of degradation. The only model measured is GPT-4o mini, so for now the numbers speak about that model alone.

How to use it

The code of the instrument, the measurement data and the passport are open, so others can rerun the procedure or check the figures. The authors describe a model's passport as the calibration of a future meter of the instruction load of prompts. That meter is a later step, not something the paper delivers.

How solid is it

The source is the paper's abstract, published on Hugging Face. The headline figures come with a sizeable sample (2958 requests, 642,430 trials), and the authors state that every number of the passport is recomputed from the published points, which makes the results checkable. The statistical model is described as yielding regular laws for every load factor, though the source gives no specifics on the shape of those laws or of the collapse of the answer.

Risks and caveats

Only GPT-4o mini is reported as measured; no other models or comparisons are given. No uncertainty interval values are given for the 37-38 and 18-19 rule figures beyond the stated ranges, and the source does not say what the full replacement density is numerically. No claim is made that the capacity figures transfer to real-world prompts or other tasks, and the meter of prompt load is described as future work. The task itself is replacing words in meaningless text, so the figures describe that setup.

“The passport of a model serves as the calibration of a future meter of the instruction load of prompts.”

— From the paper's abstract