Cartesia claims byte-level H-Nets gain up to 3-5x compute efficiency over BPE

Cartesia, writing as "we", has published the first of two posts on hierarchical networks (H-Nets), a byte-level architecture that replaces fixed tokenization with chunking learned inside the model. The authors say they developed the concept in 2025 and have since improved it significantly. The results in the post use a much improved version of H-Nets, which the authors plan to release.

The framing is Richard Sutton's 2019 essay on the bitter lesson: general methods that scale with computation and carry as few human-engineered priors as possible. The authors argue that language models still rely on hand-crafted encoding steps such as tokenizers, which hinder end-to-end learning from raw data. They believe byte-level language models are foundational for AI progress and a litmus test for general learning methods.

An H-Net chunks low-level data recursively, for example characters into words, words into phrases and phrases into concepts. Instead of pre-processing bytes into tokens, an H-Net does the chunking dynamically inside the model, and does it hierarchically and end-to-end. Standard tokenized models, and even other alternatives, use multiple training stages and chunk the data entirely offline. The architecture has an encoder, a main network and a decoder; the encoder and decoder together are the outer networks and the main model is the inner network. H-Nets can be nested, and the authors say they can work on any kind of data, including continuous data, which is hard to tokenize.

The post makes four claims. First, the authors built a standard scaling ladder for tokenized Transformers and say carefully controlled 1-stage H-Nets give compute efficiency gains of up to 3-5x purely through learned tokenization. Second, a more flexible 2-stage H-Net is said to outperform a matched open-source model with 10x less compute and data. Third, they analyze the patterns H-Nets learn, including behaviours new to language models such as in-context tokenization. Fourth, they point to ongoing research in biology and vision that shows promise for a general architecture across modalities. The second post will look at a wider range of models to explain why H-Nets are so much more efficient.

The method section describes the setup. The original H-Net paper tuned models of fixed sizes and data horizons (for example ~1.3B parameters on 100B tokens from FineWeb-Edu) and lacked compute-optimal scaling laws. For comparison, the authors cite Kimi Delta Attention and attention residuals, which measured compute multipliers of 1.16x and 1.25x. They built a BPE Transformer ladder following recipes such as Kimi and MAI, then converted each rung into a byte-level H-Net as a drop-in replacement, reusing the BPE backbone architecture and hyperparameters. This directly gave stronger validation loss (1.3x), with larger gains on subdomains such as code (2x) and out of distribution (2x on evaluation, up to 5x on out-of-domain code).

The comparison is controlled in several ways. The data mix is 70% English, 20% code, 5% math and 5% multilingual by bytes. Every sample is processed both as bytes and as BPE tokens with the same document boundaries, and loss is measured in bits per byte so tokenizers can be compared. The authors also trained a BPE tokenizer on their own corpus (128k vocabulary, matching Llama-3), which raised average token length from 4.45 bytes (Llama-3 tokenizer) to 4.57 and gave about a 1.1x compute gain on validation loss and downstream evals. They note that this means the H-Net would have an even greater multiplier against an off-the-shelf tokenizer. In tokens of that tokenizer the data is 64.7% English, 23.7% code, 6.1% math and 5.5% multilingual.

The baseline ladder uses a hidden dimension D that is a multiple of 128 with aspect ratio D/L of about 75 (L is the number of layers), multi-head attention with heads equal to layers and head dimension 128, and SwiGLU MLPs with intermediate dimension 3.6D. All runs use the Muon optimizer and a WSD learning-rate schedule. Batch size and learning rate were tuned on selected rungs and extrapolated with fitted scaling laws, and FLOPs were counted carefully, including only causal FLOPs for softmax attention so that byte-level models, which process longer sequences, are not penalized. The 1-stage H-Net comparison is deliberately narrow: it treats the H-Net as purely replacing the tokenization part of the language model.

Key facts

Cartesia says carefully controlled 1-stage H-Nets give compute efficiency gains of up to 3-5x over a tokenized Transformer, purely through learned tokenization.
A 2-stage H-Net is said to outperform a matched open-source model with 10x less compute and data.
Converting each rung of a BPE ladder to a byte-level H-Net gave 1.3x on validation loss, 2x on code, and 2x on evaluation and up to 5x on out-of-domain code.
The baseline BPE tokenizer was retrained on the authors' own corpus (128k vocabulary, 4.57 bytes per token) and gained about 1.1x compute, so the authors say the H-Net's edge over an off-the-shelf tokenizer would be larger.
The results use an improved H-Net version that the authors plan to release; a second post will explore why H-Nets are more efficient.

Why it matters

Tokenizers are one of the last hand-designed parts of the language-model pipeline. Cartesia argues that learning the chunking end-to-end fits the bitter lesson and could improve both efficiency and scalability. The post adds what the 2025 H-Net paper lacked: compute-optimal scaling laws, now a standard way to measure pretraining innovations. For scale, the authors point to Kimi Delta Attention and attention residuals, which measured compute multipliers of 1.16x and 1.25x, against their claimed up to 3-5x for 1-stage H-Nets.

Who it affects

Researchers working on tokenization, byte-level models and new architectures are the direct audience. The authors also frame H-Nets as relevant to data that is hard to tokenize or engineer for, such as science, robotics and biology, and mention ongoing research in biology and vision.

How to use it

There is nothing to download yet. The authors say they plan to release the much improved version of H-Nets used in this post, and no release date is given. The original H-Net paper and blog from last year are linked from the post. The post describes the setup in enough detail to follow the methodology: a BPE ladder with D/L of about 75, Muon, a WSD schedule and iso-FLOP sweeps.

How solid is it

This is a first-party blog post from Cartesia, and the results are self-reported. The authors describe them as evidence, not proof. The methodology looks careful: identical text and document boundaries for bytes and BPE, bits-per-byte loss, a retrained in-corpus BPE tokenizer, and causal-only FLOP counting. The text available for this retelling is cut off partway through the H-Net ladder description, so the scaling-law fits, the 2-stage results and the biology and vision results could not be checked. The matched open-source model in the 2-stage comparison is not named in the visible text. It is also not stated whether the 1.3x on validation loss is a compute multiplier or a loss ratio, or how it relates to the up to 3-5x figure.

Risks and caveats

The 3-5x figure is an upper bound ("up to") for a deliberately restricted 1-stage setting, where the H-Net is read purely as a replacement for tokenization. The 10x claim for the 2-stage model rests on a comparison with a single matched open-source model. The biology and vision work is described as ongoing and only showing promise. The results use an improved version of the architecture that has not been released, so outside replication is not yet possible. Part two, which is meant to explain why H-Nets are more efficient, is not covered here.

“If we can’t even learn language well from characters and bytes, what right do we have to make progress on the next frontier?”

— Cartesia, "From Tokens to Bytes, Part 1" (authors writing as "we")