Glonce

Hugging Face proposes small CPU fixtures to catch Transformers bugs before main

toolsThe Glonce editorsHugging Face🔥55Glonce rating55Impact35Novelty45Relevance90Surprise40Players60Credibility80

On July 30, 2026, Transformers enabled PyTorch's scaled dot-product attention (SDPA) for T5. As a side effect, UMT5's decoder could attend to future tokens, so during training it could peek at the answer it was meant to predict. Nothing crashed, and the bug stayed in main for almost two months, until a fix on September 28. When the Hugging Face team later replayed those commits, a small UMT5 model running on CPU flagged the problem on every one of them. The post is about that model and the tests built around it.

The background is scale. Transformers supports over 750 model types. Its suite has 88,000+ CPU tests on main plus 200,000+ GPU tests in a daily run, and it also covers pipelines, training, distributed execution, quantization and attention kernels. The integration tests load real checkpoints on real hardware; running them on every PR would keep many GPUs busy all day, which the authors call impractical. By the time a scheduled run finds a regression, it may already be in main.

Their answer is to shrink the model. A fixture is a small model built from the real checkpoint's configuration, with fewer and narrower layers but every kind of layer the original has, and with tokenizer and processor files unchanged. It is published once, reused, and tested on a cheap CPU runner. In the authors' CPU study the experimental UMT5 fixture used 66 MB of weights against 1,227 MB for the source checkpoint. Very large models that do not fit on standard GPU runners can also be tested this way. Transformers already has tiny-random-* test checkpoints; the authors wanted a common recipe and checks that go beyond loading a model and running a forward pass, since the UMT5 bug would have passed a forward pass.

transformers-mic is a proposed standard for packaging these tests. A Model Integration Contract (MIC) has three parts: a fixture (the shrunk model, published once and pinned); a usage path (a runnable Markdown example of something people do with the model, such as generating text or running a training step); and baselines (outputs recorded from that example and reviewed by a human). The contract holds if running the usage path on the fixture still produces the baselines. A contract covers an architecture and a usage path, not a single checkpoint, so compatible checkpoints can share a fixture while keeping their own expected results. Contracts are published in huggingface-mic. The SmolLM contract, for instance, generates from prompts such as "The capital of France is" and compares token IDs with a recorded answer; it also compares training loss, checks that gradients are finite and verifies that an optimizer step changes the weights. A fleet dataset on the Hub pins each MIC to a frozen revision along with the package versions used to record its baselines, and each CI run records the fleet revision it used. On a PR, MICs are chosen based on the files the patch changes.

In the UMT5 replay, the authors replayed frozen MICs over five months of Transformers commits. The small UMT5 fixture flagged the regression from July 30 until the September 28 fix: label log-probabilities differed from the reviewed baseline by 1.22 and training loss by about 0.20. One detail mattered. The contract used SDPA, the default attention path, and running only the eager attention variant missed the bug. A fixture only helps if it goes through the same code as the real checkpoint. The authors then reverted the fix on a proof branch in Transformers' CI; the check failed with the same differences, and the selected T5 contract's runner step took 13.4 seconds on CPU, excluding setup.

The shrink recipe has four steps. First, shrink the configuration: start from a real checkpoint and reduce hidden sizes, attention heads and depth while keeping the different layer kinds, including mixtures of sliding and full attention or dense and MoE layers. An early recipe that only cut dimensions and layers failed on mixed architectures, because two kept layers are useless if both use sliding attention and the full-attention layers vanish. Where the generic rule does not know a width, sizes are taken from the model's own ModelTester in the Transformers tests. Second, keep preprocessing unchanged: preserve vocabulary, special tokens and input dimensions, and copy tokenizer and processor files byte for byte. Third, build and validate the weights: try seeded random weights and change the inputs to check the model responds; if it does not, train it on the contract's small batch of examples; reload the saved model, check that outputs, loss and gradients are finite, and reject failed builds. Fourth, freeze and record: save accepted weights, measure repeatable outputs on a known-good Transformers build, review baselines for each supported platform and precision, and publish a pinned snapshot that CI reuses rather than rebuilding per PR. The original weights file format is kept so the conversion code that runs when users load the real checkpoint is still exercised. The Llama 4 Scout shrink report shows the effect on a big model: 108.6 billion parameters become 44.7 million, the vision stack goes from 34 layers to 2 and the language stack from 48 to 4, keeping the checked layer kinds and MoE routing style.

The team builds MICs with agents. Its CLI, tmic, creates the contract and builds the fixture. For a new model, an LLM drafts the usage code and manifest, and CPU and GPU runs measure candidate baselines. The LLM drafts; it does not approve, and changed expectations require human review. When agents hit an edge case, the authors fixed the recipe itself rather than the one fixture, with help from model experts.

On weights, a random-weight Whisper fixture barely responded to changes in the audio, making it a poor test of the audio path. The team trained it on a few tones paired with words such as "red" and "green", not to teach speech recognition but to make audio changes visible in outputs. In the weight-initialization study the random Whisper fixtures caught 18 of 24 injected faults and the overfit fixtures caught all 24. The authors say this does not settle which is better: the faults were synthetic, they have not found a real regression caught only by the overfit version, and both versions missed an assisted-decoding bug that the real Whisper checkpoint exposed. They keep random weights when those pass the checks and overfit when they need more sensitivity or readable answers. Training takes seconds but is one more step that can fail to converge and one more thing to maintain.

Open questions remain. As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs. Whether it should block depends on unmeasured things: how many real bugs it catches on PRs rather than in replays, how often it causes false alarms, and how much work baseline reviews take. The authors also do not know how many bugs, like the Whisper one, only appear with real weights. Coverage is still partial across model families and the APIs people use. When a regression slips through, they can add the missing usage path and replay it on the broken and fixed versions, and they will keep existing tests until their coverage is accounted for. They would also like to link original Hub model pages to their MICs, so a developer or agent could start from an example tested on that checkpoint with its environment and expected outputs.

Key facts

  • A July 30, 2026 change enabling SDPA for T5 let UMT5's decoder attend to future tokens; nothing crashed and the bug stayed in main for almost two months, until a fix on September 28.
  • A small UMT5 fixture on CPU flagged the regression on every replayed commit, with label log-probabilities off the reviewed baseline by 1.22 and training loss by about 0.20.
  • A Model Integration Contract (MIC) is a pinned shrunk-model fixture, a runnable usage path and human-reviewed baselines; the experimental UMT5 fixture used 66 MB of weights against 1,227 MB for the checkpoint.
  • In a study with synthetic injected faults, random-weight Whisper fixtures caught 18 of 24 and overfit ones caught 24 of 24, but the authors say this does not settle which approach is better.
  • As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs, and coverage is still partial.

Why it matters

Transformers supports over 750 model types, and running real-checkpoint integration tests on every PR would need GPUs busy all day. Scheduled runs can find a regression only after it is in main. The UMT5 case shows the cost: a silent correctness bug, with no crash and one that would pass a plain forward pass, lived in main for almost two months. The proposal is a cheap, CPU-only layer that runs earlier, with a shared recipe in place of ad hoc tiny test models. The authors are explicit that fixtures will not replace full integration tests.

Who it affects

Transformers maintainers and contributors are the first audience, since MICs are picked by the files a PR changes and run in Transformers CI. Model integrators who already build tiny test models get a common recipe. The authors also want to link Hub model pages to their MICs so a developer or an agent using a checkpoint can start from an example tested on that checkpoint, with its environment and expected outputs.

How to use it

Install with pip install "transformers-mic[runtime]" plus hf-doc-builder from its GitHub repository, run hf auth login, then run tmic run org/model --local-only --fixtures-only --no-history --bill-to '' with your own checkpoint in place of org/model. It drafts the contract, builds the fixture and records baselines on your machine, asks you to review each step, and stops before uploading anything. For a new model the draft comes from an LLM through Inference Providers, billed to your own account. Published contracts live in huggingface-mic, and the authors invite people to pick an uncovered model from the coverage space, try the tmic workflow, and contribute a usage path or review a generated contract.

How solid is it

The core evidence is a replay: frozen MICs over five months of Transformers commits, where the small UMT5 fixture flagged the regression on every commit from July 30 to the September 28 fix. A proof branch in CI that reverted the fix failed with the same differences. These are the authors' own measurements, and the post links replay results and a CI demo with reproduction details. The 66 MB versus 1,227 MB figures come from an experimental UMT5 fixture in a CPU study, and the 13.4 seconds is the runner step only for the selected T5 contract, excluding setup. The Whisper 18 of 24 versus 24 of 24 result concerns synthetic faults, not real regressions. The standard itself is described as proposed.

Risks and caveats

The MIC job currently reports failures without blocking PRs, and the authors have not measured how many real bugs it catches on live PRs, how often it triggers false alarms, or how much work baseline reviews take. Coverage is partial across model families and APIs. Fixtures can miss bugs: both Whisper versions missed an assisted-decoding bug that only the real checkpoint exposed, and the authors do not know how many bugs appear only with real weights. A fixture helps only if it goes through the same code path as the real checkpoint; running only the eager attention variant missed the UMT5 bug. Overfitting adds a step that can fail to converge and one more thing to maintain. Baselines need human review, and the LLM that drafts contracts does not approve them.

“The LLM drafts; it doesn't approve.”

— Hugging Face, Transformers Model Integration Contracts post