ngrok explains why compression is prediction

ngrok's engineering blog published an explainer on August 11, 2026 titled "Compression is prediction", arguing that data compressors and large language models are, at their core, trying to solve the exact same problem: predicting what comes next. The post builds the argument from scratch with worked numeric examples rather than asserting it outright.

It starts by distinguishing minification (stripping unneeded syntax, as in shrinking code for machines) from true compression, which relies on redundancy in the data. A worked example: a string of 9 A's, 4 B's, 2 C's, 1 D, 3 A's, then 9 D's takes 28 characters (224 bits) in standard 8-bit ASCII. Run-length encoding it as "A9B4C2D1A3D9" cuts that to 12 characters (96 bits), a 57 percent reduction.

The post then names the three components of a modern compressor: transforms (preprocessing that reshapes data to be more compressible, such as run-length encoding), models (which map each symbol to a probability based on its frequency), and entropy coders (which turn those probabilities into a final compressed bitstream).

It focuses on arithmetic coding to show how probabilities translate into compression. For the string "A B A B A A C" (probabilities: A 4/7, B 2/7, C 1/7), the piece walks through narrowing a 0 to 1 range symbol by symbol until it lands on the interval [0.38730, 0.38855), then picks 0.3876953125 as the single number representing the whole string. Encoded this way, the seven-character string needs only 10 bits, versus 56 bits in raw 8-bit ASCII. A second, more skewed string (10 A's, 1 B, 1 C, where A has probability 0.833) compresses even further: 0.82 bits per symbol on average, against 1.38 bits per symbol for the first string. The lesson drawn is that the more skewed a data's probability distribution, the better it compresses.

This average bits-per-symbol figure is entropy, which the post calls "the floor": the smallest number of bits per symbol achievable for a given set of data, and the reason there is no single universal best compressor, since entropy is specific to a particular probability distribution.

To make entropy intuitive, the post shifts to a guessing game: filling in the blank in "I saw an animal... it was a ___" using a yes/no decision tree over animals of decreasing probability (bird, squirrel, cat, fox, bear), where more probable animals need fewer guesses, i.e. shorter bit codewords. It notes this codeword assignment is itself a second entropy-coding method, Huffman coding (used in gzip and Brotli), and flags its limitation: when a probability like 0.3973 for "cat" does not split cleanly in half, the tree forces rounding, wasting bits.

The text supplied to this retelling ends there, right after introducing the basic frequency-based model (probability = count divided by total symbols), and before it draws the explicit link to next-token prediction in language models that the title promises.

Key facts

ngrok published "Compression is prediction" on August 11, 2026, arguing compressors and LLMs solve the same underlying problem.
Run-length encoding a 28-character, 224-bit example string down to "A9B4C2D1A3D9" cuts it to 12 characters, 96 bits, a 57 percent reduction.
Arithmetic coding compresses the 7-character string "A B A B A A C" (56 bits in ASCII) into the single number 0.3876953125, needing only 10 bits.
A more skewed string (10 A's, 1 B, 1 C, A at probability 0.833) compresses to 0.82 bits/symbol, versus 1.38 bits/symbol for the less skewed example, illustrating that skewed distributions compress better.
Entropy is defined as the average bits per symbol and described as the compression floor: the minimum achievable for a given probability distribution, which is why no single compressor is universally best.

Why it matters

The post's core claim, that compression and prediction are the same problem, is a foundational idea connecting classical information theory to how language models work: both a compressor's entropy coder and an LLM ultimately assign probabilities to what comes next, and better probability estimates mean better compression (or better predictions). Framing it through gzip-style tools rather than abstract theory makes the connection concrete for readers who have not seen it stated this way.

Who it affects

The piece is aimed at engineers and technically curious readers who want to understand the mechanics behind both data compression tools (gzip, Brotli) and, by extension, language models, rather than at any specific company, product or industry.

How to use it

This is a free, public educational blog post with interactive step-by-step illustrations of arithmetic coding and decoding built in; there is no product, price or license attached to the article itself. It functions as a primer for anyone building intuition about entropy coding before approaching how language models predict tokens.

How solid is it

The explanations rest on standard, long-established information theory (run-length encoding, arithmetic coding, Shannon entropy, Huffman coding) illustrated with fully worked, self-contained numeric examples that a reader can verify by hand; none of it depends on unverified claims or benchmarks. The text captured for this retelling cuts off before the post makes its explicit link between entropy coding and next-token prediction in language models, so that final connecting argument, implied by the title, is not covered here.

Risks and caveats

All the numeric examples (the A/B/C/D run-length string, the A/B/C arithmetic-coding string, the animal-guessing tree) are constructed illustrations built for teaching, not measurements from real files or production compressors, and gzip and Brotli are named only as examples of tools that use these techniques, without benchmarks or version details for either.

“The most important thing to understand about entropy is that it's the floor. This is the smallest number of bits per symbol we can achieve for a given set of data.”

— ngrok, "Compression is prediction"