Glonce

VocabX tokenizer cuts a Qwen3-4B essay grader's inference cost 77%

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact40Novelty60Relevance95Surprise45Players5Credibility55
VocabX tokenizer cuts a Qwen3-4B essay grader's inference cost 77%

A blog post on Hugging Face previews results from the VocabX paper, which its authors say they will publish soon. The paper is still under review, and the post warns that the numbers may change before publication. The method rebuilds the vocabulary of a fine-tuned small language model around the cost of each token rather than its frequency.

The test case is an automated grader for Portuguese essays that the authors run in production. It is built on Qwen3-4B (4 billion parameters), takes the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency. It still used the original Qwen3-4B tokenizer, built for many languages and kinds of text, even though the grader works in one language and one response format. The authors name two costs. The text the grader writes on every request, such as the same JSON envelope, competency names and evaluation phrases, took more tokens than needed. And at every generated token the model scored a vocabulary of 151,669 entries, many of which the task almost never uses.

The post argues that counting tokens is not the same as counting the bill. Inference has two phases: prefill, which processes the whole input in one pass and is limited by compute, and decode, which generates one token at a time and is limited by memory bandwidth. In the grader case the authors measured a generated token at 7.4 times the cost of a token read. They say the ratio varies with hardware, problem, batch size and context length and can be larger, but that the direction holds in general: generating costs more than reading. The output layer (LM head) adds a third cost. It holds one row per vocabulary entry, takes almost 10% of the parameters in a model the size of Qwen3-4B, and is read in full for every generated token.

The authors say existing methods, namely fast vocabulary transfer (FVT), multi-word tokenization (MWT) and Liquid AI's in-place tokenizer expansion (in-place BPE), pick tokens by frequency, not cost, and each works on a single layer. VocabX instead weights each eliminated token by the price of the field it appears in (system prompt, user message or model response). In the grader, a token saved in the response is worth 7.4 times one saved in the input. It works on two layers under one cost criterion, each with its own budget: sub-word pieces, and multi-word spans turned into single "macro tokens", including whole lines and JSON envelopes. To find those spans it uses five miners, each specialised in one kind of repetition: long words, whole lines, expressions, complete sentences, and structures with punctuation such as JSON and tags. Averaged over eight corpora, the punctuation miner contributed the most, and on code it alone accounted for 17.5 points. Without all five, token reduction on the grader would drop by 55.6 points.

The third idea is estimating what a document sample cannot see. Measuring each candidate's remaining gain exactly would mean reprocessing the whole corpus after every choice. Sampling is the usual shortcut, but a candidate absent from the sample then gets zero gain. An expression that appears 37 times in 50,000 documents shows up in a 400-document sample only about 26% of the time, and in a contracts corpus only 16.6% of candidates appear in such a sample. VocabX counts a candidate's total saving exactly and estimates only the part not yet covered by already chosen tokens, using the behaviour of the candidates the sample does see. With the same sample and budget, that change alone raised token reduction on a contracts corpus from 44.7% to 52.8%.

For the grader, the authors trained seven versions with the same base model, data, recipe and seed, differing only in tokenizer. Each ran the same 500 requests on one machine with an RTX 5090 GPU and the vLLM server, and cost was calculated from measured runtime. The average response fell from 350 to 101 tokens and cost fell 77.0%. The bench score, which combines grading accuracy and the accuracy of screening off-topic or rule-breaking essays, rose from 0.703 to 0.810. The output layer shrank from about 152,000 to about 65,000 rows.

Getting there took a correction. The first vocabulary fused each competency's grade with the surrounding JSON into one macro token. It was the most compact (a 73.3% cost cut) but the bench score fell to 0.689, below the production grader's 0.703. The authors explain that each grade-and-structure combination became a different variant of the same token, so the model chose both in a single step. Giving the grade its own token cost 2 extra tokens per response, lifted the score to 0.810 and was, in the measurement, even cheaper at 77.0%. Their lesson: compression has a cost when the compressed span carries the decision the model exists to make.

To check whether the gain was specific to the grader, they compared VocabX with FVT, MWT and in-place BPE on five public domains (contracts, medical papers, scientific papers, news and Python code) and three model families (Qwen3, Llama-3.1, Gemma-3), with the same mining and test documents and the same vocabulary ceiling. VocabX compressed more than the strongest reference method in all fifteen combinations, by 1.2 to 9.3 points. In estimated inference cost, the pruned version also led in all fifteen, with average margins of 10.2 points on Qwen3, 7.5 on Llama-3.1 and 13.6 on Gemma-3. On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, against 61.8% for the best in-place BPE configuration.

Key facts

  • On a Portuguese essay grader built on Qwen3-4B, VocabX cut measured inference cost by 77.0% while the bench score rose from 0.703 to 0.810.
  • Average response length fell from 350 to 101 tokens, and the output layer shrank from about 152,000 to about 65,000 rows.
  • The method weights tokens by cost, not frequency: in the grader a generated token costs 7.4 times a token read, so savings in the response count for more.
  • A first version that fused each grade into a JSON macro token cut cost 73.3% but scored only 0.689; giving the grade its own token fixed quality.
  • Across five public domains and three model families, VocabX compressed more than the strongest reference method in all fifteen combinations, by 1.2 to 9.3 points.

Why it matters

Most teams treat the tokenizer as a fixed property of the base model. The authors argue it is one of the biggest cost levers for a model specialised in one domain, and it is set before any training step. The post's central point is that the token count is not the bill: generating a token costs more than reading one (7.4 times in their grader), and every unused vocabulary row adds data to move at each decode step. Choosing tokens by cost, on both sub-word and multi-word layers, is what separates VocabX from FVT, MWT and in-place BPE as the authors describe them.

Who it affects

The post targets teams that fine-tune general open models such as Qwen and Gemma for a single domain or task and serve them in production, especially small models served with moderate batches, where the authors say the decode effect is strongest. Examples of repetitive output named in the post include JSON envelopes, contract boilerplate and code identifiers.

How to use it

The text does not describe a released tool, so there is nothing to install. What it offers is a recipe. Weight each token by the price of the field it appears in. Mine repeated spans with several specialised miners. Keep the vocabulary small. Check that compression does not swallow tokens that carry the model's actual decision, such as the grade in the grader. The authors trained each variant with the same base model, data, recipe and seed, so the tokenizer was the only change.

How solid is it

This is a preview from the authors' own blog of a paper that is still under review, and the authors say the numbers may change before publication. The 77.0% figure comes from one production grader (Portuguese essays, Qwen3-4B), seven versions of it, and 500 requests on one RTX 5090 with vLLM, with cost calculated from measured runtime. The wider comparison covers fifteen combinations of five public domains and three model families, but those results are stated as token reduction and estimated inference cost, not as the 77% figure. No independent verification is mentioned.

Risks and caveats

The 7.4 times ratio is specific to the measured grader case, and the authors say it varies with hardware, problem, batch size and context length. The first VocabX version shows the main risk: the most compact vocabulary lowered quality (0.689 versus 0.703) until the grade got its own token. VocabX does not put LM-head size in its selection criterion; it controls vocabulary size through the sub-word merge and a fixed macro budget. Absolute dollar cost, latency and throughput figures are not given, and the post does not say the production grader has switched to the new vocabulary.

“The goal is not the cheapest possible vocabulary, but the cheapest one that preserves the tokens that matter.”

— VocabX blog post, Hugging Face